Your Agent Re-Sends the Whole Conversation Every Turn

Here is a number from Anthropic’s own documentation that is worth sitting with. A request containing the system prompt "You are a scientist" and the message "Hello, Claude" counts 14 input tokens. The same endpoint, given a one-line message and a single trivial get_weather tool, returns 403.¹

Nothing about the conversation got longer. A tool definition arrived, and with it the system prompt the API adds whenever any tool is present — 286 tokens on Claude Opus 5.² That is the smallest possible version of the thing that dominates agent bills: you are billed for the shape of the request, not for the part you typed.

The Messages API is stateless. There is no session on the server holding your conversation. Every turn, your client sends the tools, the system prompt, and every message so far, and the model reads all of it again. A 20-step agent session is not 20 small requests. It is 20 requests of steadily increasing size, and the total grows quadratically.

This post is about where those tokens actually sit, and about the three server-side mechanisms that attack them. They do genuinely different things, and the most interesting result below is that one of them, left at its defaults, made a realistic workload 75% more expensive.

I covered how prompt caching works mechanically in an earlier post on prefill and decode — prefix hashing, breakpoints, the 1.25× write premium. This one assumes that and looks at what happens across turns.

What one request actually contains

A request is rendered in a fixed order — tools, then system, then messages — and splits cleanly into two parts with opposite behaviour.³

The prefix (tools plus system prompt) is byte-identical on every turn. The tail (messages) grows monotonically: each user message, each assistant reply, each tool result is appended and never removed.

That split is the whole map. Caching attacks the prefix and the already-seen part of the tail. Context editing attacks specific blocks inside the tail. Compaction replaces the tail wholesale.

The prefix tax: what tools cost before anyone says anything

Tool definitions are pure prefix. You pay for them on turn 1 and on turn 200, whether or not the model calls a single one.

Two things are worth pulling out of that chart.

The first is that tool_choice is not free. Switching from auto to any costs 120 extra tokens on Opus 5, every request, forever.²

The second is the gap between the top rows and the bottom ones. A handful of hand-written tools costs a few hundred tokens. Declaring the browser toolset costs about 6,610 — and enabling all four optional members adds roughly 880 more.² Over a 40-step session that one decision is over 264,000 billed input tokens before the agent does anything.

This is the cheapest optimisation available and almost nobody does it: stop shipping tools the agent will not use in this session. Both toolsets accept a configs field to disable members, and the docs note that turning off zoom alone removes about 410 tokens from every request.²

What replay costs

Now the tail. Here is the worked example used for the rest of this post — a research agent that reads documentation.

One request per step, Claude Opus 5 rates: $5.00/MTok base input, $6.25 for a 5-minute cache write, $0.50 for a cache read, $25.00 output.² Everything below is arithmetic on those published numbers — it is an illustration of the shape, not a measurement of your workload.

The uncached line is not a straight line, and that is the entire point. Each step is bigger than the last, so cost per step keeps rising: step 1 costs $0.05, step 12 costs $1.45. Caching flattens it to roughly constant per-step cost, because the only thing paid at full rate is the new material.

Three mechanisms, three different targets

Caching does not make the context smaller. The tokens are still there, still counted against the context window, still read by the model — they are just cheaper. Two other mechanisms actually remove content.

flowchart TD
R["One request"] --> A["Prefix<br/>tools + system"]
R --> B["Tail<br/>messages + tool results"]
A --> C["Prompt caching<br/>0.1x on a hit"]
B --> C
B --> D["Context editing<br/>drops old tool results"]
B --> E["Compaction<br/>replaces history with a summary"]

Context editing is surgical. The clear_tool_uses_20250919 strategy removes old tool results from the conversation once input crosses a threshold, keeping the most recent few. Its defaults: trigger at 100,000 input tokens, keep at 3 tool uses, clear_at_least unset, clear_tool_inputs false — so by default the model still sees what it asked for, just not what came back.⁴ A companion strategy, clear_thinking_20251015, drops thinking blocks from earlier turns; when you combine the two it “must be listed first in the edits array.”⁴

Compaction is wholesale. At its default trigger of 150,000 input tokens (minimum allowed: 50,000), the API summarises the conversation and returns a compaction block. You pass that block back, and “the API automatically drops all content blocks prior to the compaction block, continuing the conversation from the summary.”⁵

The rationale the docs give for editing is not primarily financial: “Context is a finite resource with diminishing returns, and irrelevant content degrades model focus.”⁴ Worth remembering when the cost numbers below look unflattering.

The bill context editing hands you

The context-editing docs contain one sentence that turns out to be the most expensive thing on the page: tool-result clearing “invalidates cached prompt prefixes when content is cleared.”⁴

Of course it does. Caching is a prefix hash. Delete a tool result sitting near the front of the message list and every byte after it has moved, so the cached prefix ends where the deletion starts. What used to be a cache read at $0.50/MTok becomes a cache write at $6.25/MTok — 12.5× more per token.

So I ran the worked example three ways: no editing, editing at the documented defaults, and editing with clear_at_least set to 100,000.

The defaults line never catches up, even extending the simulation to 60 steps. The mechanism is simple: once the context sits just above the trigger, every step pushes it over again, so every step clears one result and pays a full cache write on everything after it. You have swapped a $0.04 read for a $0.50 write, permanently.

This is precisely what the docs advise against, in a sentence easy to read past: “clear enough tokens to make the cache invalidation worthwhile. Use the clear_at_least parameter to ensure a minimum number of tokens is cleared each time.”⁴ The trigger decides when clearing is allowed. clear_at_least decides how often it actually happens, and that is the parameter that moves the bill.

But cost is not the only axis, and on the other one the defaults look very different. Without editing, this session’s context reaches 287,950 tokens by step 12 and passes a million around step 40. With editing at defaults it plateaus at about 88,000 tokens and stays there — one cleared result per step, forever. That configuration is not trying to save money. It is keeping the session alive and the model focused, and it succeeds completely.

Notice also what the plateau does to the third mechanism: context held below 100,000 never reaches compaction’s 150,000 trigger. The two features are not additive. Tune the first and you may have decided you will never use the second.

The compaction cost that isn’t in input_tokens

Compaction has a trap of its own, and it is a reporting one.

Summarising a 180,000-token conversation requires a model call over those 180,000 tokens. That call is billed. But the top-level usage fields do not include it — they “reflect the sum of all non-compaction iterations,” and to get the real number you must “sum across all entries in the usage.iterations array.”⁵

So a dashboard reading usage.input_tokens will show compaction as a dramatic saving, because the expensive part has been moved into a field nobody is summing. In the documented example the top-level report is 23,000 input tokens while the compaction iteration alone consumed 180,000 input and 3,500 output.⁵

Measuring instead of guessing

Every claim above is visible in your own responses. Four fields, worth logging permanently:

u = response.usage

# Caching. If cache_read is 0 across requests that share a prefix,
# something upstream is changing and you are paying full price.
u.cache_read_input_tokens
u.cache_creation_input_tokens

# Context editing. Empty means your trigger never fired —
# which, at a 100,000-token default, is most conversations.
response.context_management.applied_edits
# -> [{"type": "clear_tool_uses_20250919",
#      "cleared_tool_uses": 8, "cleared_input_tokens": 50000}]

# Compaction. The top-level totals EXCLUDE the summarisation pass.
total = sum(i.input_tokens for i in u.iterations)

For anything you have not sent yet, /v1/messages/count_tokens accepts the same request shape — system prompts, tools, images, PDFs — and is “free to use,” on rate limits independent of message creation.¹ It is the right way to price a tool definition before committing to it: count the request with the tool and without, and subtract.

Two caveats. The count “is an estimate” and may differ slightly from what you are billed, and it deliberately ignores cache_control, since “prompt caching only occurs during actual message creation.”¹ It tells you how big a request is, not what it will cost.

What this costs you

Caching loses money on short conversations. The write premium is real and paid upfront; in the model above, turn 1 with caching costs more than turn 1 without. Break-even came at step 3 for a 5-minute write and would come later for the 2× one-hour write. A one-shot classifier should not have caching on.

Context editing trades cache hits for context space, and the exchange rate is 12.5:1. Configured carelessly it is a straightforward cost increase. That can still be the right call — if the alternative is a session that dies at the context limit or a model distracted by forty stale page dumps, paying more per step is the cheap option. Just know which one you bought.

Clearing tool results is lossy in a way summarisation is not. With clear_tool_inputs false, the model keeps seeing that it fetched a page and no longer sees what the page said. For a search-and-synthesise agent that is fine. For one that must reconcile details across many documents, it silently removes the evidence while leaving the impression that the work was done.

Compaction is lossy too, and it costs a full pass over the context to be so. Every compaction is another inference over your longest prompt. Trigger it often and the summarisation bill can exceed what you saved — and by default you will not see it in the field you are watching.

Trimming the prefix is the only lever with no downside. Fewer tools is fewer tokens on every request, cached or not, plus a smaller decision space for the model. It is the first thing to try and usually the last thing anyone does.

Takeaways

• Statelessness is the cost model. Every turn re-sends everything; a session’s billed input grows quadratically with its length. In the example above, 287,950 unique tokens were billed as 1,775,700.
• Caching changes the price, editing and compaction change the contents. They are not alternatives — and tuning the second one hard enough means the third never runs.
• clear_at_least matters more than trigger. The trigger says when clearing is permitted; clear_at_least says how often you pay the cache-write penalty. At defaults, this workload got 75% more expensive.
• Read usage.iterations, not usage.input_tokens. With compaction on, the top-level totals omit the most expensive call in the request.

References