How often do AI code edits fail? Ours failed 5.8% of the time

Ask how often an AI coding agent's edits fail and you get a shrug, a vibe, or a figure someone half-remembers from a thread. That is odd, because it is trivially countable: the edit either landed or the tool refused it, and the refusal is written down in the trace with its reason attached. So we counted ours. On 2026-07-28, across 129 preserved traces, our agent issued 434 str_replace edits and 25 came back as errors.

What the number measures

A str_replace edit is the surgical option: instead of rewriting a file, the agent quotes the exact text it wants replaced along with its replacement, and the tool swaps them. If that anchor cannot be found, or turns up in more than one place, the tool refuses, writing nothing and returning an error saying which. That refusal is the failure we counted, so it costs a retry rather than a corrupted file.

Two things it is not. Not correctness: an edit can apply cleanly and still be the wrong change. And not a benchmark of the field — one agent, one harness, one set of traces.

The 25 failures, by cause

Sorted by the error returned, the whole tail fits in three rows:

The split is the useful part. The anchor causes are dead even at 11 apiece — all but three of the 25 between them — and both are failures of quoting rather than of reasoning: the agent knew the change it wanted and described the target badly, once too vaguely to be unique and once simply wrongly. Almost nothing here is the model misunderstanding the task, which is what people picture when they imagine an edit failing.

Why the rate is as high as it is

This is the part worth taking away, because none of it is exotic. Three cheap mechanisms sit in the loop already, each aimed at a cause in that table.

• A whitespace-tolerant match. If the anchor differs from the file only in indentation or trailing spaces, the edit still applies, removing a whole category of near-miss. The surviving error reads “even ignoring whitespace” — the tolerance announcing it was already tried.
• A re-read nudge when an anchor misses. A failed match comes back with a prompt to go and look at the file again, turning the commonest self-inflicted failure into one wasted turn rather than a loop.
• Explicit guidance to disambiguate. The agent is told to include the line above and below its target. A three-line anchor is far likelier to be unique than a one-line one — the direct remedy for the “found N matches” row.

None of that needed a bigger model or a second agent. It is error-message design and prompt guidance, applied to the path an edit takes when it goes wrong.

The decision this number drove

Here is the part that makes measuring worth the effort, and it is not a boast. Our 2026-07-28 technique scan put Microsoft's SWE-Edit top of the shortlist. Microsoft's report describes lifting edit success on open reasoning models from a 75.6–82.0% band to 93.9–95.9%, using a viewer/editor subagent split plus a whole-file-rewrite fallback. A large, well-evidenced gain — and we did not adopt it.

Read the rows together and the logic is plain. SWE-Edit's gain is almost entirely the distance from the low band to the high one, and we start inside the high one. Its benefit comes from lifting models off a floor we are not standing on, while the costs stay real: more model calls per edit, and a fallback that rewrites a file wholesale where three lines would have done.

None of that was knowable without the baseline. With no number the shortlist item reads as a straightforward upgrade, you adopt it, and you pay for a gain you cannot collect. The most useful result of measuring your own system is finding out which improvements cannot help you.

How to decide

Weighing a technique like this, three questions do most of the work:

• What is your own number? Count the tool refusals in your traces before reading anyone's benchmark. A headline gain means nothing until you know which end of its range you stand on.
• Does the gain come from lifting a floor you are on? Most improvements are measured on systems failing in one specific way. If yours does not fail that way, the gain is not transferable, however sound the study.
• What does it cost if it lands neutral? Price the extra model calls, the added latency and the new failure modes, then assume the benefit is zero. If your own count sits in Microsoft's pre-fix band rather than near ours, start with the three cheap fixes above instead.

Frequently asked questions