What is prefix caching (KV cache reuse)?
- Prefix caching, also called KV cache reuse, is the trick of keeping the key/value tensors a transformer already computed for a prompt and reusing them when a later prompt starts with the same tokens. Only the differing suffix gets prefilled.
- It requires an exact token-level prefix match — identical token ids, in order, from position zero. One changed token at position 5 invalidates everything from position 5 onward.
- Done right it is the difference between re-reading the whole conversation on every turn and reading only what's new. In our runtime, fixing it took steady-state time-to-first-token from 18.5s down to 1.39–4.71s.
Prefix caching is trivial to describe and unreasonably easy to break. The concept fits in a sentence. The implementation fails silently, degrades gradually, and every direct assertion you write to test it will pass while it's broken. I know because I shipped three separate bugs that each killed it stone dead, in the same runtime, in the same month. Here's the mechanism and the failure modes.
The KV cache, in one section
When a transformer processes a prompt, it computes a key tensor and a value tensor for every token at every layer. That collection is the KV cache. There are two phases and they have completely different cost profiles.
Prefill is computing those tensors for the prompt you were handed. It's compute-bound and it scales with prompt length — longer prompt, more prefill work, and the attention term grows faster than linearly, so it gets worse as the context gets long. Decode is generating new tokens one at a time, appending each one's keys and values to the cache as it goes. Decode is memory-bandwidth-bound and is dominated by how many tokens you generate rather than by how long the prompt was, though a longer cache does cost bandwidth on every step.
| Prefill | Decode | |
|---|---|---|
| What it does | Computes K/V for every prompt token | Generates tokens one at a time, appending to the cache |
| Bottleneck | Compute | Memory bandwidth |
| Scales with | Prompt length | Tokens generated |
| What the user feels | Time to first token | Tokens per second |
| Can prefix caching help? | Yes — this is the entire point | No |
So prefill is the part that punishes you for long prompts, and prefill is exactly the part prefix caching eliminates. In a chat app the prompt grows every turn while the genuinely new content stays tiny. Without reuse you pay for the whole history again on every message. With reuse you pay only for what changed.
The rule: exact token match, from position zero
If a new prompt begins with the same tokens as one you already processed, the KV entries for that shared span are still valid. Nothing about those positions changed, so their keys and values didn't either. You keep them and prefill only the suffix.
The requirement is strict and it's the whole story: identical token ids, in order, starting at position zero. Not similar. Not semantically equivalent. Not "the same system prompt with today's date swapped in." A single differing token at position 5 poisons every position after it, because each of those positions attended to the token you changed. That's why prefix caching is fragile — it needs token-level stability from template code nobody thinks of as performance-critical.
Failure mode 1: ranking, not matching
Our lookup() returned the longest stored match, gated on n <= len(prompt_ids). Looks fine. It isn't. An entry covering the entire prompt satisfies that gate and wins for being longest — and leaves an empty suffix. The caller had a reasonable-looking branch for "can't generate with an empty suffix," so it threw the cache away and paid a full prefill. Meanwhile the genuinely useful entry, a 2,328-token system prefix that would have left a cheap ~104-token suffix, lost the ranking for being shorter.
The fix was one character of intent: require n < len(prompt_ids) so the suffix is never empty. Steady-state time-to-first-token went from 18.5s to 1.39–4.71s, roughly 4–14x.
The debugging lesson matters more than the bug. Every direct check passed. Key equality: True. Common prefix: 2,328. Types: correct. A hash check or a key comparison cannot reveal a ranking bug, because nothing about the keys is wrong — the wrong correct answer is being chosen. Dumping every candidate entry's gate decision and hit decision found it in a single run.
Failure mode 2: template drift
With thinking disabled, the chat template appended an empty <think></think> block to the generation prompt, but rendered historical assistant turns without it. Read that twice: the prompt for turn N was therefore never a prefix of the prompt for turn N+1. They diverged four tokens before the end of the stored prompt — stored length 5,923, live length 7,380, common prefix 5,919.
Reuse: zero. Cost: about +11.6s per step, growing linearly with conversation length, which is exactly the shape of "the assistant gets slower the longer you talk to it."
There's a second lesson buried in this one. There were two prompt-building call sites. I patched one, re-ran, and the divergence dump came back byte-identical — which is how I found out about the other. If a fix produces literally no change in the diagnostic output, you patched something that isn't on the path.
Failure mode 3: a mutating system block
A memory feature spliced recalled facts into the system message — the very top of the prompt — using the most recent user message as the recall query. In a chat that's harmless. In an agent loop, every tool result becomes the new most-recent message, so the recalled text changed on every step and everything after it diverged. Measured divergence: token 314 of a 4,748-token prompt — so roughly 93% of the prompt was recomputed every step, because a block near the top kept moving.
The fix was to recall against the first user message instead. That made the block stable, and it's a better query anyway: a tool result full of code is mostly noise for a semantic lookup.
The design rules that transfer
These were bugs in one runtime, and I won't pretend they're universal. The rules that came out of them are the transferable part.
- Order the prompt by mutation rate. Most stable first — system prompt, tool definitions — and most volatile last. This is the single decision that determines how much of your prompt is cacheable at all.
- Never inject anything per-request above something you want cached. Timestamps are the classic offender. A current-time string in the system prompt makes literally every request a cold prefill, and it will never show up as an error.
- Make cache-miss paths loud. A silent
exceptaround cache reuse converts a correctness-preserving regression into an invisible performance cliff. Nothing breaks. It just gets slower, forever. - Instrument the near-miss, not just hit/miss. Log the longest common prefix and what came next on each side.
common=5919 of 5923localizes a bug thathit=Falsenever will.
What it looked like in practice
| Failure mode | Where it diverged | Effect |
|---|---|---|
| Ranking (longest match wins, suffix empty) | Nowhere — the match was correct, the choice wasn't | Full prefill every request; 18.5s TTFT |
| Template drift (empty think block) | Last 4 tokens; common 5,919 of 5,923 | Zero reuse; ~+11.6s per agent step, growing |
| Mutating system block (memory recall) | Token 314 of 4,748 | Everything after token 314 recomputed every step (~93% of the prompt) |
The through-line: in all three cases the model was fine, the weights were fine, the math was fine. What looked like "the local model is slow on long conversations" was prompt assembly quietly breaking a token-identity guarantee three different ways. If you're building an inference stack, the cache is probably not where your bug is. The code that builds the string you hand the cache is.
Questions about prefix caching (KV cache reuse)
Does prefix caching change the model's output?
No. The requirement is an exact token-level match, and for an identical run of tokens the key and value tensors are still valid — reusing them gives the same numbers recomputing them would. Prefix caching skips work, it doesn't approximate it.
Why doesn't a similar prompt hit the cache?
Because similarity isn't the criterion. A hit needs identical token ids, in order, from position zero. Rewording a sentence, changing whitespace, or swapping a date all produce different token ids, and everything from the first differing position onward has to be recomputed.
Where should the volatile parts of a prompt go?
At the end. Order the prompt by mutation rate: the system prompt and tool definitions first because they rarely change, then the conversation, then whatever changes on every single request. Anything that varies per request — a timestamp, a retrieved snippet, a tool result — invalidates every token that follows it.
How do I tell a cache miss from a near miss?
Log the longest common prefix between the stored prompt and the live one, plus the next token on each side. A boolean hit=False tells you nothing about why. A line reading common=5919 of 5923 tells you the divergence is four tokens from the end of the stored prompt and points straight at the template.
Try Outlier free
Free tier is Nano 4B + Lite 9B, local and private, no account. Pro is $20/mo or $149/yr for all 7 model tiers including Plus 397B. Lifetime Pro from $99 (Founding 200) or $200 (Founders 500). Apple Silicon only.
Download for Mac