Prefix cache latency benchmarks — Outlier v659–v662

Between v659 and v662 I found three separate bugs that all had the same symptom: the prefix cache existed, looked healthy, and never got used. Every one of them made long conversations get slower and slower. Here are the raw numbers from before and after each fix, the method, and the earlier benchmark these numbers throw out.

Table 1 — single-request TTFT, steady state (v659)

v659 was the prefix-cache ranking fix: the lookup was selecting the wrong cached entry, so the cache went unused and the whole prompt was re-read from scratch. Nothing about the cache looked broken from the outside — the only place it showed up was the timings.

Net: roughly 4–14x faster steady-state TTFT. The first request is still the slow one; later requests mostly land near 1.4 seconds, though the identical-message run still shows an occasional 3.7 s outlier — see the third value in row 2. Regression check: multi-turn continuation still reused turn 1 at n=2400 with a 120-token suffix.

Table 2 — six-step agent loop with growing tool results (v660)

v660 was the chat-template fix, and the ugly one. The template left turn N and turn N+1 diverging at the very end of the prompt, and a divergence anywhere is a total miss. Reuse was zero, and every agent step re-read the entire conversation.

Before the fix, each step cost about 11.6 seconds more than the one before it, growing linearly with conversation length. After, it's flat: 16.1 s at step 3 and 16.0 s at step 6. That's 4.2x at step 6, and the gap keeps widening with every additional step, because one side grows and the other doesn't. Note step 1 got slower — 8.4 s to 10.7 s. I'm leaving that in the table rather than hiding it.

Table 3 — five-step agent loop, native tool-calling (v662)

v662 was the memory-recall fix. Recall was changing the prompt from one turn to the next, so the prefix broke again. Divergence showed up at token 314.

After the fix, steps 3 through 5 sit flat at 24.3, 25.9 and 25.5 seconds while the prompt itself grows from about 7,000 to 9,200 tokens. Flat latency against a growing prompt is the whole point. The cache ended the run with six entries and hits at n=384, n=2506, n=6989 and n=7149, against a final length of 9,231 tokens.

The residual ~25 s per step is mostly decoding about 160 tokens, not prefill. Prefill is what the cache removes; generation cost stays put. Look only at the headline seconds and you'd assume the fix barely helped — which is why the divergence instrumentation matters more than the stopwatch here.

Method and caveats

• Hardware: Mac Studio M1 Ultra (Mac13,2), 64 GB unified memory, macOS. One machine, one configuration. No cross-vendor comparison is claimed or implied.
• Software: Outlier v659 through v662, measured on the installed signed binary — not a dev build, and not a synthetic micro-benchmark. These are runs under normal use.
• Agent-loop steps include model decode time, so they are not pure prefill measurements. Table 3's discussion above breaks that out.
• Divergence figures come from the runtime's own longest-common-prefix instrumentation, not from inference about timings.
• All figures are first-party measurements by Matt Kerr, July 2026.

What these numbers invalidate

An earlier internal SWE-bench run scored 0 out of 5 and reported per-step degradation from about 30 seconds to 73 seconds. I read that as a model capability problem. It wasn't.

That run measured these three bugs. The per-step degradation is the same linear growth as Table 2, and the failures came from an agent loop that got slower until it was useless. Any conclusion it supported about how well the local model reasons is void. It has to be re-run on v662 or later before it means anything, and until then I'm not quoting the number. Publishing the correction is the point of this page.

Why prefix caching matters at all

A model re-reads the whole conversation before writing a single new token. In a chat that's mildly annoying. In an agent loop, where each step appends a tool result and calls the model again, step 6 re-reads everything from steps 1 through 5. A prefix cache keeps the computed state for the shared beginning so only the new tail gets processed. When it works, per-step latency goes flat. When it silently doesn't — what all three of these bugs did — it grows forever, and nothing in the logs looks broken.

Questions about these latency measurements