My Blog's Correction Ranks #15. The Mistake It Fixed Ranks #6.

“Agents Don’t Need Memory, They Need Documentation” went to the top of Hacker News on October 3, 2026, 96 points and 59 comments in, arguing that vector-database “memory” for AI agents has a structural flaw: a similarity search over stored snippets “ranks how close two snippets are in embedding space. That’s it.” It cannot tell a correct memory from a stale one, because nothing in a cosine similarity score encodes correctness or recency.¹ The post names five failures this causes and proposes structured documentation instead. What it does not include is a benchmark. No numbers, no retrieval test, nowhere the claim gets to fail in public.

This blog has something most test corpora don’t: two real, dated, published corrections. Twice this project shipped a number, found it wrong, and fixed it in place with a dated note. That is a ground truth a synthetic test set cannot easily manufacture, and it was sitting in this repository’s own git history the whole time. So I built the retrieval system the critique describes, pointed it at this blog’s 25 published posts, and asked it the kind of question an agent with “memory” would actually need to answer.

What “agent memory” usually means

A vector-based memory system works in three steps. First, text (a conversation, a document, a past decision) gets split into chunks. Second, each chunk is converted into an embedding: a list of a few thousand numbers produced by a model trained so that semantically similar text produces numerically similar lists. Third, when an agent needs to recall something, its question gets embedded the same way, and the system returns whichever stored chunks have embeddings closest to the question’s, measured by cosine similarity, a number between -1 and 1 that captures how aligned two vectors are regardless of their length.

Nothing in that pipeline checks whether a chunk is still true. A sentence saying “X is 34,040” and a later sentence saying “X is actually 31,484” can both describe the exact same fact, phrased similarly enough that their embeddings land close together, with no vector-space signal for which one supersedes the other. That is liao.gg’s argument. I built the pipeline exactly as described and tested it against two cases where I already knew the right answer, because I was the one who wrote the correction.

flowchart LR
  S["25 published posts<br/>(.mdx source)"] --> C["Chunker<br/>captions, labels,<br/>paragraphs"]
  C --> E1["924 chunks"]
  E1 --> V["Gemini embedding-001<br/>RETRIEVAL_DOCUMENT"]
  V --> DB["924 stored vectors"]
  Q["A question"] --> V2["Gemini embedding-001<br/>RETRIEVAL_QUERY"]
  V2 --> R["Cosine similarity<br/>against all 924"]
  DB --> R
  R --> Rank["Ranked list,<br/>most similar first"]

Building a corpus with a known answer

I split every post’s source into 924 chunks: figure captions, alt-text labels, and markdown paragraphs, stripped of code fences and raw SVG, keeping diagram descriptions since those carry real claims too.² Every chunk was embedded with Google’s gemini-embedding-001, using the RETRIEVAL_DOCUMENT task type the model documentation specifies for indexed content, producing the model’s default 3,072-dimensional vector.³

Two of those 924 chunks have a correct ranking I can state in advance, because this blog corrected them itself, on the record, with a dated note:

• September 17. A post estimated a container image build at “three orders of magnitude” slower than a git worktree checkout. The next day’s measurement found it was 364 times slower, not roughly 1,000. The caption was rewritten in place.⁴
• September 20. A post called 34,040 tokens the “fixed charge” for a subagent to exist. The next day’s measurement found that figure was boot plus one tool-calling turn combined; the boot alone is 31,484. Rather than rewrite the sentence, this one got a dated correction blockquote appended directly below it.⁵

Two corrections, two different repair strategies: one rewrote the claim, the other appended a note next to it. Both are real content, sitting in the actual corpus, not constructed for this post.

Case one: a nine-rank gap

For the September 20 correction, I asked the retrieval system three differently-phrased versions of the same question and recorded where the stale chunk and the correcting chunk each landed, out of all 924:

The most natural phrasing of the question, “what is the fixed token cost of a subagent before it does any work,” is the one where it matters most: the stale claim ranked 6th, the correction that explicitly supersedes it ranked 15th. The other two phrasings flipped the order, correction ahead of mistake, by similar margins. There’s no consistent winner here, which is itself the finding: nothing about embedding distance reliably favors the current fact over the one it replaced. It depends on how closely your question’s wording happens to match one version’s wording, a property that has nothing to do with which version is true.

Reading the live post explains part of why the stale number keeps winning. I counted every place on that one page stating 34,040 as the fixed cost, uncorrected: the figure’s caption, its alt-text label, two lines of text baked directly into the diagram’s SVG, two nodes inside a Mermaid flowchart, and four separate body sentences.⁵ That’s ten uncorrected restatements of the same number, two of them inside diagrams nobody redrew when the text was fixed. Against that, exactly one paragraph says otherwise. A retrieval system doesn’t know any of those ten are describing the same now-outdated fact; it just sees ten chunks near the query and one chunk slightly less near it.

Case two: nobody wins, and a third number muddies it further

The September 17 correction is messier in an instructive way. The old caption (“three orders of magnitude”) and the new one (“364×, measured in the follow-up post”) are nearly the same sentence with one key phrase swapped, which is exactly the scenario liao.gg’s critique predicts breaks similarity search worst: two versions close enough in wording that embedding distance barely separates them. It didn’t. Across three phrasings, the old and new captions landed within one to two ranks of each other every time, both near the top, with no reliable order between them.

That’s already the point: a correction and the mistake it fixes are, to a cosine similarity score, practically the same memory. But there’s a complication the retrieval system surfaced that I didn’t design for. “364×” describes a one-time container image build. A different, also-true number from the same follow-up post, “a warm container start is about 3×, not 364×,” describes a different thing entirely: restarting an already-built container. Both are correct. They answer different questions. And for some phrasings of “how much slower is a container than a worktree,” the 3× chunk ranked ahead of both the right and the wrong version of the 364× answer.

The failure liao.gg describes isn’t just “stale beats current.” It’s broader: similarity search has no mechanism for knowing a question needs disambiguating in the first place. It will hand back whichever stored number sounds most like the question, whether that number is wrong, right-but-answering-something-else, or right-and-exactly-what-was-asked, with the same confident absence of a signal either way.

Why a single document doesn’t have this failure mode

liao.gg’s proposed fix is structural, not algorithmic: keep a small set of living documents an agent reads before it works and updates after, rather than an ever-growing pile of retrieved snippets.¹ The reason this sidesteps the ranking problem isn’t that documents are smarter than embeddings. It’s that the question “which version is current” stops being a retrieval problem at all. In a correctly maintained document, there is exactly one row for “the subagent’s fixed boot cost,” and updating it means overwriting that row, not appending a competing one somewhere a similarity search might or might not surface it.

This blog already runs a version of that discipline for the person steering it: a local, gitignored file logs every published topic with a one-line summary, explicitly marked “already covered, no need to look again,” rather than leaving that judgment to whatever a search happens to surface.⁶

Trade-offs worth naming

• This is a small, specific corpus. 924 chunks from 25 posts is nowhere near the scale of a production memory system running for months, and the claim being tested is about a structural property of cosine similarity, not about scale, which is why a small, verifiable corpus is still a fair test. But a bigger corpus could show different margins.
• One embedding model, not the field. Google shipped a newer gemini-embedding-2 after I ran this; a reranking step on top of raw similarity, which several production RAG systems use, might catch exactly the kind of near-duplicate collision these two cases produce. I tested the baseline the criticism is actually about, not every mitigation available.³
• The correction didn’t always lose. Two of six query results favored the current fact over the stale one. The honest claim is not “vector memory always prefers mistakes,” it’s “vector memory has no reliable mechanism for preferring the correction,” and a coin that sometimes lands on the right side is still not a mechanism.
• Documentation has its own cost. The single-source-of-truth approach only works if someone (or some agent) actually does the “update” step in “prompt, consult, build, update.” This blog’s own two corrections happened because a person read the follow-up measurement and went back to fix the first post. That discipline is a process cost a passively-indexed vector store never asks you to pay, and also never does for you.

Takeaways

• A cosine similarity score carries no notion of correctness or time. Two versions of the same fact, one superseding the other, can and did land one to two ranks apart, functionally tied, because the only thing the score measures is how similar the wording is.
• The stale version can win by sheer repetition. Ten places on one page restate an uncorrected number against one paragraph that corrects it; a retrieval system has no way to know those ten chunks are all describing a single fact that already changed.
• Ambiguous questions pull in true-but-wrong-context answers, not just wrong ones. A correctly measured number describing a different comparison outranked both the right and wrong versions of the number actually being asked about, for some phrasings.
• A maintained document sidesteps the question by construction, not by being smarter. When there is exactly one place a fact can live, “which version is current” stops being something retrieval has to guess.
• An uncited claim is still testable, even without the author’s own numbers. liao.gg’s post shipped no benchmark; the ground truth to check it against was already sitting in this project’s git history.

References