A model can be better at coding and worse at coding
- On chat coding (the "write me this function" kind) our 26B Quick tier scored 21/23 against our 27B Code tier's 20/23, in 37% less time.
- On agentic coding, meaning fix a real bug in a real repository unaided, the same Quick tier scored 0/30. Not a low score. Zero.
- Both numbers are ours, measured on the same Mac against the shipping build.
- Pick a coding model from a chat-coding benchmark and you can pick one that cannot do the job at all.
We measured our own tiers twice, expecting the second run to confirm the first. It did the opposite, and the gap was large enough to change which model we ship as the default for agent work.
The two measurements
Chat coding, n=23. Self-contained problems, one prompt each, graded on whether the code is correct. Quick (26B MoE) got 21. Code (27B) got 20. Quick was 37% faster. More interesting than either score: the failures did not overlap at all. Between them the two models solved every problem in the set.
Agentic coding, n=30. SWE-bench: a real issue in a real repository. The model has to work out which files matter, produce a patch, and have the project's own tests pass. Quick scored 0. It was not producing wrong patches. In 29 of 30 it produced no usable patch at all, stalling during discovery, calling tools without ever converging on an edit.
Why the same model does both
The two tasks look alike and are not. Chat coding asks a model to write code once the problem is already stated. Agentic coding asks it to work out what the problem is: which file, which function, which of forty plausible candidates. Then make one small correct change and stop. The second is mostly navigation and restraint. Fluency at writing code contributes surprisingly little.
This is not a claim that one of our models is better. It is a claim that "good at coding" is not a single property, and one benchmark cannot tell you which kind you bought.
What we changed
Code stays the default for agent and repository work despite losing the chat-coding comparison, because the agentic gap is not close. Quick stays excellent at what it is good at. We would rather ship the model that finishes the job than the one that wins the demo.
Method, so you can discount it properly
- Chat coding: n=23, our own problem set, graded on correctness, shipping build, M1 Ultra. Small n, so treat a one-point difference as noise. The zero failure overlap is the finding, not 21-vs-20.
- Agentic: SWE-bench Verified, n=30, blind (no test patch, no hint which tests should pass), official Docker harness. Our Core tier on the same harness scores 23/50 = 46.0% (2026-08-09, build 1.11.757, roughly ±14pp at that n).
- Agentic scores are harness-dependent. A scheduling fix in our own runner moved a tier from 11/30 to 15/30 with no change to the weights. Compare within a harness, never across.
We publish the numbers we measured, including the ones that make our own lineup look inconsistent, because a benchmark that only ever flatters the product is not a measurement. Outlier runs these models on a Mac, offline. The FAQ covers what each tier is for, and support is one person.