Why AI gets slower the longer you talk to it

You open a chat in the morning and it's quick. Forty messages later every reply takes a beat too long, then two beats, and you wonder whether the model got worse or your Mac is tired. Neither. It's doing more work per answer than it was this morning, for a reason that's easy to explain and hard to fix.

The model re-reads the whole conversation, every single time

Here's the part that surprises people: a chat model doesn't remember your conversation. Nothing persists between turns. When you send message twenty, the software hands it everything — your first message, its first answer, all eighteen turns in between — as one block of text, and it reads that block from the top.

That reading pass is called prefill, and it's why nothing appears on screen for a moment after you hit enter. The model has to ingest the whole prompt before it can produce the first token.

The arithmetic is unforgiving. Turn 2 means re-reading two turns; turn 20 means re-reading twenty. The prefill grows with the conversation, and the pause before the first word gets longer. That's not degradation. That's the design.

The fix is a prefix cache, and it's all-or-nothing

Nobody accepts that cost, so every serious runtime has a mitigation: the prefix cache, sometimes called KV cache reuse. Turn 20's prompt starts with exactly the same text as turn 19's — same conversation, more stuck on the end. If the runtime kept the work it did for turn 19, it can reuse that and only process the new part. When it works, latency stops depending on conversation length.

The catch is the word exactly. Reuse requires the earlier prompt to be a byte-identical prefix of the later one. Not similar. Identical. And the top of a prompt is where software likes to inject things — a timestamp, a memory block, a system instruction rebuilt slightly differently. Any drift up there throws away the entire cache. Prefix caching has its own explainer if you want more depth.

Which is why the symptom is confusing from outside. A working cache and a broken one look identical until you time them. You just notice it was snappy this morning and it's crawling.

Three bugs, one symptom: what I found in July 2026

I'll use my own app as the worked example, because I have the numbers. Over one debugging session across Outlier v659 to v662 I found three separate causes of the same symptom. Each time I fixed one I thought I was done, and another was hiding behind it.

That last row is the whole article in one line: the prompt got roughly 30% longer and the time barely moved. A flat line instead of a ramp.

99.9% matching is worth exactly the same as 0% matching

The v660 bug is the one I'd print on a poster. 5,919 tokens out of 5,923 matched — 99.9% — and reuse was zero. Not reduced — zero.

Prefix reuse doesn't degrade gracefully — there's no partial credit for being close. The runtime walks forward from token one, and the instant it finds a mismatch everything after it is unusable, because every later token's state depends on the ones before it. Four bad tokens at the tail of a 6,000-token prompt cost the same as a completely different prompt.

That's also why these bugs survive normal debugging. Every check I had was passing — keys matched, hashes matched, the cache had entries. It was perfectly healthy and simply never used.

What you can do about it, on any AI you use

You can't patch someone else's cache, but you can shorten the prompt. That works everywhere:

• Start a fresh conversation for a new task. The most effective move by far. A new thread is a short prompt, and a short prompt is a short prefill.
• Don't edit early messages mid-thread. Rewriting message three changes the top of the prompt and invalidates every cached token after it. You've paid for the whole conversation again.
• Don't paste bulk text you don't need. That 4,000-line log you dropped in at turn two isn't a one-time cost — it's in the prompt for every remaining turn.
• Notice the shape of the slowdown. If the wait grows with thread length and resets when you start over, it's prompt length. If it's random, something else is wrong.

Cloud services have the same problem — you just can't see it

None of this is specific to running models on your own machine. The structural fact is the same everywhere, and a hosted service has the same fragility at the top of the prompt.

The difference is what you can do about it. Their cache runs on their servers: you can't see whether it's hitting, and you can't fix it when it isn't. I could dump every cache entry and every hit-or-miss decision until the ranking bug fell out. That's a narrow advantage of local software — it doesn't make the model smarter, it just puts the person who can fix the plumbing in the same room as the problem.

So my claim is deliberately narrow, because it's the one I measured: after v659 through v662, latency in Outlier no longer grows with conversation length. Closely related: why AI forgets your conversations — same root fact, different consequence.

Frequently asked questions