Hermes's Memory Search Beat Embeddings, but Only Through Its Last-Resort Fallback

“Agent memory” is one phrase for three different mechanisms. One is a short file of notes that sits in the system prompt. Another is a search box over old conversations. The third is a service that reads your conversations with a second LLM and writes down conclusions about you. Hermes Agent, the open-source personal agent from Nous Research, ships all three.¹

I read how each one works in the source, then measured the two that run locally.² The test set is LoCoMo, a public benchmark of ten long chat histories with 1,536 questions that each point at the exact turn holding the answer.³ The results were not what I expected going in. The keyword search beat an embedding model, but it only works at all because of a fallback that fires for almost every query.

The three places memory can live

Before the numbers, here is where each mechanism sits relative to the model.

flowchart LR
F[("MEMORY.md 2,200 chars<br/>USER.md 1,375 chars")] -->|"frozen at session start"| P["System prompt"]
H[("Session DB<br/>SQLite FTS5")] -->|"agent calls session_search"| T["Tool result:<br/>raw messages"]
X["External provider<br/>e.g. Honcho<br/>(LLM calls)"] -.->|"optional, one at a time"| P
P --> M["Model"]
T --> M

The built-in notes are two files with hard caps. The store defaults to 2,200 characters for MEMORY.md and 1,375 for USER.md.⁴ Both are copied into the system prompt as a frozen snapshot when a session starts, so a write mid-session reaches disk but not the prompt until the next session.⁵ The reason is the prompt cache: a prompt prefix that never changes can be reused cheaply on every turn.

Search is a separate tool. Its own docstring says it makes no LLM calls: every result is an actual stored message.⁶ Underneath is SQLite’s FTS5 full-text index. Optionally you can add one external provider, and the manager rejects a second one.⁷ Honcho is the documented example.

The notes: 3% of a conversation

A 2,200-character cap is small. How small depends on what you ask it to hold, so I measured against LoCoMo. Each conversation averages 72,676 characters of dialogue (about 18,900 tokens with the o200k_base tokenizer, across 588 turns). The cap is 3.0% of that.

LoCoMo also ships a list of short facts per conversation, written by an LLM, one per evidence turn. They are a fair stand-in for what a diligent agent would write into its notes. Those facts total 21,698 characters per conversation, 9.9 times the cap.

I then gave the notes the best case I could build. A greedy “oracle” that already knows every question picks the facts that unlock the most questions per character and stops at 2,200. It fits 27.5 facts on average. Those facts let it answer 469 of the 1,536 questions, or 30.5%.

That 30.5% is generous. A real curator does not know tomorrow’s questions, and my facts are written one per turn, in a different voice than a person would use for a notes file. Still, 2,200 characters is roughly 430 tokens, which is cheap to carry on every turn. When the file fills up, the add operation fails and tells the model to remove or shorten something in the same batch.⁸ Curation is a continuous editing job, and the editor is the same model that is trying to finish your task.

The search: it only works because of one retry

For search I replayed the code path in hermes_state_search.py against the same data. Each of the 5,882 dialogue turns is one indexed message, and each question is sent as a query. The table is declared without a tokenizer option, so it uses SQLite’s default unicode61, which does not stem.⁹ Hits are ranked by BM25, SQLite’s standard relevance score (rarer words count for more).¹⁰

The detail that matters is FTS5’s treatment of several words in one query. They are joined by an implicit AND, so a message must contain every word.¹⁰ A natural-language question almost never matches that way. I sent all 1,536 questions as typed:

• With only the AND query and the trigram fallback, 99.8% of questions returned nothing. Only three of the 1,536 returned any rows at all, and none of them held the evidence.
• With Hermes’s last step, an OR retry that matches any word, 98.6% of questions (1,514) were answered by that retry.

That last step is in the source with a comment that describes this exact failure: a paraphrased query “misses a stored sentence that lacks even ONE word.”¹¹ So the maintainers know. It is also the only reason the tool returns anything for a question phrased as a sentence.

With the retry, the evidence turn is in the top 10 for 54.4% of questions and in the first slot for 24.9%. Multi-hop questions, which need more than one turn, reach 38.3% at top 10.

A natural worry is that an agent would write a better query than the raw question. I tried a stand-in: the same questions with stopwords removed. It moved top-10 from 54.4% to 58.1%, and even then 1,483 of 1,536 queries still needed the OR retry. This is a proxy, not an agent, so treat the gain as a rough bound. Trimming filler words helps by a few points. It does not change which path does the work.

Lexical search beat a small embedding model

An embedding model turns text into a vector and ranks by similarity, which is supposed to survive rephrasing. So I ran the same questions through bge-small-en-v1.5, a small quantized embedding model, over the same turns.¹² It reached 46.1% at top 10, behind the keyword path’s 54.4%.

The overlap between question words and evidence words explains most of the gap. I split the questions by what fraction of their keywords appear in the evidence turn:

Keyword search finds almost nothing when no words are shared, about half the time when a few are, and nearly always when a third or more are. The embedding model is flatter: a little rescue when words differ, a lower ceiling when they match. The two fail in different places. Of the 1,536 questions, 307 were found only by keywords, 180 only by the embedding model, 528 by both, and 521 by neither. Taking the union of the two covers 66.1% at top 10, a bigger jump than the 3.7 points that stripping stopwords added.

The 521 that neither found are a puzzle. They lean toward questions that need more than one turn (33%, against 24% of the rest), but their evidence turns are about as long as everyone else’s (median 157 characters against 172). I do not have a clean explanation for what they share.

The third architecture, which I did not measure

Honcho is a separate memory backend that Hermes talks to through a plugin. After a conversation turn it can run an LLM pass that derives conclusions about the user, then inject a summary of them into the prompt.¹³ I did not run it. The setup has you create an API key, and the quality of its conclusions depends on a model I cannot inspect.

What the documentation does let me compute is the cost shape. The defaults are a context fetch every turn and an LLM call every second turn, with 1 to 3 passes per call.¹⁴ Over 100 turns that is 100 API fetches and 50 LLM calls at default depth. At the maximum depth of 3, called every turn, it is 300. That is arithmetic on documented defaults, not a measurement. The point is only that this is the one mechanism whose bill grows with conversation length even when the user asks nothing about the past.

What this does not show

• LoCoMo is two people chatting, not a person talking to an agent. Hermes sessions hold tool calls and long outputs, and the index also covers tool names and call arguments, which my replica does not. Recall on real sessions could be better or worse.
• Hit at k is necessary, not sufficient. Finding the evidence turn does not mean the model answers correctly. I did not run an LLM to grade answers.
• The default tool returns fewer results than ten. It defaults to three, with a window of messages around each.⁶ The top-1 and top-5 numbers are closer to what an agent sees.
• The embedding model is small and quantized. A larger one would do better. I chose one that runs on a laptop CPU, the kind of thing a personal agent would use, and I did not tune it.
• The oracle is greedy. It is not provably optimal, though it only needs to be roughly right to show the cap is the constraint.
• The released LoCoMo file differs from its paper’s abstract. The paper reports about 300 turns and 9,000 tokens per conversation; the file I measured averages 588 turns and 18,902 tokens.³ I report my own counts.

Takeaways

• The notes file is a 2,200-character budget, about 3% of one long conversation. Even an oracle that knew every question fit only 30.5% of the answers.
• Search with an AND query returned nothing for 99.8% of questions typed as sentences. Hermes works because a final OR retry rescues 98.6% of them.
• Writing keywords instead of a sentence added about 4 points at top 10 (54.4% to 58.1%) in my stopword-stripping proxy.
• A small embedding model reached 46.1% against 54.4% for keywords, but it found 180 questions the keywords missed, and the union covers 66.1%.
• The search is strong when words are shared and weak when they are not. If your agent’s users paraphrase heavily, add a semantic layer instead of tuning the lexical one.
• The external provider’s cost scales with turns, not with questions. It makes LLM calls on a cadence even when the user asks nothing about the past, so check the cadence settings before enabling it on a long-running agent.

References