Your LLM Is Two Different Machines

Here is a question with a genuinely interesting answer: why do LLM APIs stream?

The convenient explanation is that it feels nicer. The real one is that the model cannot do anything else. Generating the hundredth token requires having generated the ninety-ninth, so the tokens arrive one at a time whether or not you display them that way. Streaming is not a UX choice layered on top of the model — it is the model’s execution shape leaking through the API.

That shape has a name. Inference runs in two phases, prefill and decode, and they behave so differently that treating them as one workload is the root of most confusion about LLM latency, cost and context limits.

Two phases, opposite bottlenecks

Prefill processes your entire prompt at once. Every input token is known upfront, so the model can push all of them through the network in parallel. NVIDIA describes it as “a matrix-matrix operation that’s highly parallelized,” and it “effectively saturate[s] GPU utilization.”¹

Decode produces output tokens one at a time, each conditioned on everything before it. There is exactly one new token to process per step, so the same computation becomes “a matrix-vector operation that underutilizes the GPU compute ability compared to the prefill phase.”¹

That is the whole asymmetry, and it has a sharp consequence. Prefill is compute-bound: the GPU’s arithmetic units are the constraint. Decode is memory-bandwidth-bound: the GPU spends most of its time waiting for weights and cached state to arrive from memory, with the arithmetic units mostly idle.¹

Once you have this split, a pile of unrelated-looking observations collapse into one explanation. A long prompt delays the first token but not the ones after it. A long output costs time roughly linearly, regardless of prompt size. Batching helps throughput enormously but makes any single request slower to start. All three are the same fact seen from different angles.

The KV cache is the thing that actually fills up

Decode is memory-bound partly because of the weights, and increasingly because of the KV cache.

Attention needs the key and value vectors for every previous token. Recomputing them at each step would be quadratic and absurd, so they are computed once and kept. NVIDIA gives the size directly:¹

KV cache bytes per token = 2 × num_layers × (num_heads × dim_head) × precision_in_bytes

The leading 2 is the K and the V. Everything else is fixed by the architecture, which means the cache grows linearly with context length and there is nothing clever you can do about the slope — only the constant.

That constant is what grouped-query attention attacks. Multi-query attention collapses all key-value heads into one, which is fast but “can lead to quality degradation”; GQA uses “an intermediate (more than one, less than number of query heads) number of key-value heads” and reaches “quality close to multi-head attention with comparable speed to MQA.”² It is not a minor tuning knob:

This is the honest answer to “why is the context window that size and not bigger.” The window is not a number someone picked. It is where the KV cache stops fitting next to the weights at a batch size that still makes economic sense.

Why batching is not optional

If decode is bandwidth-bound, the fix follows mechanically. The expensive part of a decode step is reading the weights, and that read is the same whether you are generating one token or thirty-two.

The catch is that batching in the obvious way — collect n requests, run them together, return them together — wastes most of the benefit, because sequences finish at different times and the whole batch waits for the slowest. Production servers use continuous batching instead, admitting and retiring sequences at the granularity of a single decode step.

Doing that requires solving a memory problem. Every sequence’s KV cache grows unpredictably, so reserving a contiguous block per sequence means reserving for the worst case and wasting the rest. vLLM’s PagedAttention borrows virtual memory paging: the KV cache lives in fixed-size blocks that need not be contiguous. The paper reports “near-zero waste in KV cache memory” and 2–4× throughput at the same latency versus the prior state of the art.³

flowchart LR
R["Request<br/>arrives"] --> C{"Prompt prefix<br/>already cached?"}
C -- "Yes" --> S["Skip prefill<br/>for that prefix"]
C -- "No" --> P["Prefill<br/>compute-bound"]
S --> Q["Scheduler<br/>continuous batching"]
P --> Q
Q --> D["Decode step<br/>memory-bound"]
D --> K["Paged KV cache"]
K --> D
D --> V["Speculative decoding<br/>draft + verify"]
V --> O["Tokens<br/>stream out"]

When prefill and decode fight

Continuous batching creates a scheduling problem the two-phase split makes inevitable. A decode step is tiny; a prefill for a 100K-token prompt is enormous. Put them on the same device and the long prefill occupies the GPU while every in-flight generation stalls — one arriving request adds a visible stutter to everyone else’s stream.

Chunked prefill is the fix: split a large prefill into pieces and batch those pieces alongside decode steps, so generation keeps advancing. vLLM prioritises decode requests and batches pending decodes before scheduling prefill work.⁴

It is a redistribution, not a free win, and the direction is exactly what you would predict. Smaller chunks protect inter-token latency for sequences already generating, at the cost of time to first token for the request being admitted; larger chunks do the reverse.⁴ The knob does not remove the conflict between the two phases — it just decides which request absorbs it.

Speculative decoding attacks the same bottleneck from another direction. A small draft model proposes several tokens; the large model verifies them in a single parallel pass — which it can do cheaply, because verification looks like prefill rather than decode. The original paper reports 2–3× acceleration “with identical outputs,” and this matters more than the speedup: the sampling scheme is constructed so the output distribution is unchanged.⁵ You are not trading quality for latency. You are exploiting the fact that the GPU was idle anyway.

Prompt caching: the part you actually control

Everything above happens inside someone else’s serving stack. Prompt caching is the one lever exposed to you, and it makes sense only in terms of prefill: if a prefix has been prefilled before, its KV cache can be reused instead of recomputed.

The pricing shows how large the saving is. On Claude Opus 5, base input is $5.00/MTok, a 5-minute cache write is 1.25× that, a 1-hour write is 2×, and a cache read is 0.1× — $0.50/MTok.⁶

The mechanism is a prefix match, and that word does the work. Caching is keyed on a cumulative hash of everything up to a breakpoint, rendered in the order tools → system → messages.⁶ Any byte that changes anywhere in the prefix changes the hash, and everything after it is recomputed.

This produces the single most common way caching silently fails: putting the breakpoint after something that varies.

# Broken — the timestamp changes every request, so the prefix hash does too.
# You pay the 1.25x write premium on every call and never get a hit.
system = [
    {"type": "text", "text": LONG_STABLE_CONTEXT},
    {"type": "text",
     "text": f"Current time: {datetime.now()}",
     "cache_control": {"type": "ephemeral"}},
]

# Correct — the breakpoint sits on the last block whose prefix is identical
# across requests. Volatile content moves after it, uncached.
system = [
    {"type": "text", "text": LONG_STABLE_CONTEXT,
     "cache_control": {"type": "ephemeral"}},
]
messages = [{"role": "user", "content": f"Current time: {datetime.now()}\n{query}"}]

Three further details decide whether caching works at all:

• There is a minimum. Prefixes below the model’s threshold are not cached even when marked — 512 tokens on Claude Opus 5, up to 4,096 on other models.⁶ Below it, cache_control is silently a no-op.
• You get four breakpoints. Automatic caching consumes one of them.⁶
• The TTL runs from the start of the request, not the end. A response that streams for four minutes leaves about one minute of a five-minute window for the follow-up.⁶

Verification is one line and worth wiring into your logs permanently: if usage.cache_read_input_tokens is zero across requests that share a prefix, something is invalidating it.

What this costs you

Every technique above trades something. Being specific about what separates engineering from cargo-culting.

Throughput and latency are in tension, structurally. Larger batches mean more tokens per weight read and a longer wait for any given request to be scheduled. There is no batch size that optimises both, which is why serving stacks expose it as a knob rather than choosing for you — and why a p99 latency target and a cost-per-token target are negotiating with each other, not merely coexisting.

The advertised context window is not a promise about your batch. A million-token window is a statement about what the model can attend to, not about how many concurrent million-token sequences a deployment can hold. The KV cache is the binding constraint, and it is per sequence. Long-context and high-concurrency are the same budget spent twice.

Prefill and decode want different hardware. One is starved for arithmetic, the other for bandwidth. Running both on the same device means at least one is being under-served at any moment, and that they interleave badly under load — chunked prefill manages the collision but cannot remove it. This is why disaggregating the two phases onto separate pools is now a serious deployment pattern rather than an exotic one.

Speculative decoding is a bet, and losing costs real compute. When the draft model’s proposals are rejected, the verification pass is discarded work. On workloads where the draft model predicts poorly — unusual domains, heavy tool-call syntax, low-temperature outputs it was not trained on — acceptance rates fall and the technique can be net negative. It pays off on predictable text and degrades quietly on the rest.

Cache writes cost more than no caching at all. A 1.25× write that never gets a hit is a 25% surcharge, and a 2× one-hour write that never gets a hit is worse. Caching is only a saving if the prefix is genuinely reused within the TTL — so the traffic pattern, not the prompt size, decides whether it is worth enabling.

Takeaways

• Prefill and decode are different workloads on the same hardware. One is compute-bound and parallel; the other is memory-bandwidth-bound and sequential. Almost every LLM performance question resolves to which one you are asking about.
• Prompt length buys time to first token; output length buys everything after. They are separate budgets and should be optimised separately.
• The KV cache grows linearly with context and cannot be made sublinear — GQA changes the constant, not the slope. It, not the advertised window, is what limits concurrency.
• Batching is how decode is made economical, and it works precisely because the weight read is fixed cost. That is also why it raises per-request latency.
• Prompt caching is prefill reuse, matched on an exact prefix. One varying byte early in the prompt turns the whole thing into a surcharge. Log cache_read_input_tokens and watch it.
• Speculative decoding is free latency only when the draft model is right. Measure acceptance on your own traffic before assuming the published speedup.

References