Can a local 27B model code like Claude? The measured answer

"Can a small local model really code like Claude?" is the question I get most, and the honest answer is neither the hype ("small models are as good as frontier now") nor the dismissal ("a 4-bit 27B is a toy"). It's a specific, measurable gap that I've reverse-engineered into a roadmap. This is that roadmap: the measured starting point, how far pure local engineering carries it, how far training carries it, and the three walls that a local 27B does not cross — with the numbers behind each claim. No vibes.

The measured starting point: ~45% on SWE-bench Verified

SWE-bench Verified is the benchmark that actually matters for coding agents — real GitHub issues with hidden tests that either pass or don't. Run blind against a slice of it, the local 27B agent resolved 18 of 40 tasks — about 45%. That number is measured, not projected, and it comes with the honesty this whole strategy is built on:

• It's a blind result — the agent didn't see the hidden tests.
• It's not a leaderboard number. 40 tasks is a real, honest slice, not the full 500-task Verified set. Treat it as a measured signal, not a headline rank.
• The single-shot baseline (one attempt, no scaffold) sits far below this. Most of the ~45% is the agent loop — localization, tool calls, running tests, iterating — not the raw model.

The roadmap: what closes the gap, and what doesn't

Here's the honest ladder, from the measured 45% up to the wall. Each rung is a real, distinct source of gain — and the last rung is a ceiling, not another rung.

Stage	What it adds	Measured / target band
Blind baseline	The agent loop as it stands today	~45% (18/40, measured)
Local engineering	Execution-oracle selection, self-verification, verbatim memory, whole-file edits, grammar-constrained tool emission — no new training	mid-50s to low-60s
Training	LoRA + distillation on the target distribution	~70–80% in-distribution
The wall (Claude's band)	The last ~15–20 points — a genuine ceiling for a 4-bit 27B, not a tuning knob	Claude 65–72%

Read that carefully. Local engineering — the part Outlier owns and ships — is worth roughly ten points on its own, with no model change at all. Training buys another big jump, but only in-distribution: it makes the agent great at the kind of work it was trained on, not universally smarter. And then there's a wall.

The three hard walls

This is the part most "local AI" writeups skip, because it's the part that doesn't flatter the local model. There are three failures a local 27B does not engineer its way out of. They are structural.

Wall	What it is	Why it's structural
1. Per-token slip floor	Exact enumeration, counting, arithmetic — the "get every one of these 40 items exactly right" tasks	A structural property of a 4-bit 27B. Quantization noise sets a floor on per-token precision that no scaffold removes.
2. Breadth of world & library knowledge	The long tail of "I just need to know this obscure API / edge case / library quirk"	A parameter-count wall. Distillation transfers style, not the fact long-tail — you can't distill knowledge the student has no room to store.
3. Exact recall of a far-back byte-span	Reproducing a specific span verbatim from deep in a long context	Attention over a long window is lossy at small scale; the exact bytes from far back get approximated, not recalled.

These three are why the wall exists. Claude clears all three because it has the parameters and the precision budget to. A 27B doesn't, and I'm not going to pretend it does.

Where a local 27B already matches Claude

Now the flip side, and it's a real one. On bounded, in-distribution, testable tasks, the local 27B is already there. This session, running autonomously and verified by running the tests, the local agent built:

Build	Result (verified by running the tests)
Bank Account class with overdraft protection	13/13 tests passing
5-file library management system	Correct borrow/return + edge cases
Contacts app, sqlite-backed	Persists and queries correctly
Arithmetic expression evaluator	Precedence-correct: 2+3*4=14, (2+3)*4=20
33-file autonomous build	Every file compiles

These aren't cherry-picked one-liners; they're multi-file programs with real edge cases, and they pass the tests. On this class of work — the class most day-to-day coding actually is — you would not feel a downgrade from Claude.

And there's a quieter property that matters more than any single build: it's truthful. When a build's tests fail, the agent reports the failure honestly instead of declaring a false success. A coding agent that lies about passing is worse than useless; one that tells you the truth about where it stands is one you can actually delegate to.

Where it never matches Claude locally

• Breadth-bound tasks — the ones that hinge on knowing an obscure fact or library quirk (wall #2).
• Flawless multi-step novel reasoning with no test to check against — no execution oracle can save you when there's nothing to run.
• 40-plus-step autonomous runs — error compounds over a long horizon; small models drift where Claude holds the thread.

The biggest lever: an execution oracle

If you take one thing from this page, take this. The single highest-leverage change is how you select the answer. There are three ways to pick which of a model's attempts to trust, and they are wildly different in value:

Selection method	What it actually checks	Trustworthy?
The model approves itself	Whether the model thinks it's right	No — a confident wrong answer looks identical to a right one
"It compiles"	Syntax, not behavior	Weak — compiling code is routinely wrong
Execution oracle (run the tests)	Whether the behavior is actually correct	Yes — the ground truth

Selecting on run-the-tests is what turns a small model's noisy best-of-N attempts into a reliable pass. The model doesn't have to be right on the first try — it has to be right on one try, and the oracle finds it. This is the lever behind most of the local-engineering jump in the roadmap above, and it's the one Outlier leans on hardest.

Self-verification: the agent checks its own work

The newest piece extends the oracle inward. When a test fails, the agent now computes the true answer itself and fixes whichever side is actually wrong — the code, or a test that asserts a wrong value. Crucially, there's a guard that blocks weakening a good test to cheat the check. That guard is the whole game: an agent allowed to edit tests will, if you let it, "fix" a failure by deleting the assertion. Blocking that is what keeps self-verification honest instead of a loophole.

The honest takeaway

Can a local 27B model code like Claude? On bounded, in-distribution, testable work — yes, measurably, today. Across the whole surface — no, and there's a real ~15–20-point wall with three structural causes I won't hand-wave away. The interesting truth is how much of the gap is engineering rather than raw capability: roughly ten points sit in selection, memory, and edit strategy that don't touch the model at all. That's the seam a local coding agent lives in — private, offline, free of usage caps, and honest about where it stands. When the task hits a wall, the right move isn't to pretend; it's to escalate. Everywhere else, the local model earns its keep.

Frequently asked questions