Can a local 27B model code like Claude? The measured answer
- Not across the board, and I can tell you exactly where. A local on-device 27B coding agent (the Core tier, running in MLX on an M1 Ultra) scored a measured 46% on SWE-bench Verified: 23 of 50 blind.
- Local engineering with no new training is the lever we are working on (a target, not a measured result; the full stack measured 46% on a separate 100-task sample, directional, see how local AI gets smarter): execution-oracle selection, self-verification, verbatim memory, whole-file edits, and grammar-constrained tool emission.
- Training (LoRA + distillation) is a further lever. Its in-distribution gain is a target, not a measured result. Claude Code is still faster and stronger on the hardest work.
- It solved the bounded, testable tasks we set it (a bank account class passed 13/13 tests), and it is truthful: it reports failure honestly instead of claiming a false pass.
- The single biggest lever is an execution oracle: select on running the tests, not on "it compiles" and not on the model approving its own work.
Outlier Pro: $39 once · Black Friday sale: $16.99 once through December 2. Then $39.
Free to start (Nano and Lite). macOS 15+, Apple silicon. Refund window: 30 days, no questions asked. Priced in US dollars. Checkout shows your currency and any local tax.
"Can a small local model really code like Claude?" is the question I get most, and the honest answer is neither the hype ("small models are as good as frontier now") nor the dismissal ("a 4-bit 27B is a toy"). It's a specific, measurable gap that I've reverse-engineered into a roadmap. This is that roadmap: the measured starting point, how far pure local engineering carries it, how far training carries it, and the three walls that a local 27B does not cross, with the numbers behind each claim. No vibes.
The measured starting point: 46% on SWE-bench Verified
SWE-bench Verified is the benchmark that actually matters for coding agents. Real GitHub issues with hidden tests that either pass or don't. Run blind against a slice of it, the local 27B agent resolved 23 of 50 tasks: 46%. That number is measured, not projected, and it comes with the honesty this whole strategy is built on:
- It's a blind result, the agent didn't see the hidden tests.
- It's not a leaderboard number. 50 tasks is a real, honest slice, not the full 500-task Verified set. Treat it as a measured signal, not a headline rank.
- The single-shot baseline (one attempt, no scaffold) sits far below this. Most of the 46% is the agent loop: localization, tool calls, running tests and iterating, not the raw model.
The roadmap: what closes the gap, and what doesn't
Here's the honest ladder, from the measured 46% up to the wall. Each rung is a real, distinct source of gain, and the last rung is a ceiling, not another rung.
| Stage | What it adds | Status |
|---|---|---|
| Blind baseline | The agent loop as it stands today | 46% (23/50, measured) |
| Local engineering | Execution-oracle selection, self-verification, verbatim memory, whole-file edits, grammar-constrained tool emission: no new training | Target, not measured (the full stack measured 46% on a separate 100-task sample, directional) |
| Training | LoRA + distillation on the target distribution | Target, not measured |
| The wall (Claude's band) | The remaining gap: a ceiling for a 4-bit 27B, not a tuning knob | No measured figure on this page |
Read that carefully. Local engineering, the part Outlier owns and ships, is the lever we are working on, with no model change at all (the full stack measured 46% on a separate 100-task sample, so I have not shown a ten-point gain). Training buys another big jump, but only in-distribution: it makes the agent great at the kind of work it was trained on, not universally smarter. And then there's a wall.
The three hard walls
This is the part most "local AI" writeups skip, because it's the part that doesn't flatter the local model. There are three failures a local 27B does not engineer its way out of. They are structural.
| Wall | What it is | Why it's structural |
|---|---|---|
| 1. Per-token slip floor | Exact enumeration, counting, arithmetic, the "get every one of these 40 items exactly right" tasks | A structural property of a 4-bit 27B. Quantization noise sets a floor on per-token precision that no scaffold removes. |
| 2. Breadth of world & library knowledge | The long tail of "I just need to know this obscure API / edge case / library quirk" | A parameter-count wall. Distillation transfers style, not the fact long-tail. You can't distill knowledge the student has no room to store. |
| 3. Exact recall of a far-back byte-span | Reproducing a specific span verbatim from deep in a long context | Attention over a long window is lossy at small scale; the exact bytes from far back get approximated, not recalled. |
These three are why the wall exists. Claude clears all three because it has the parameters and the precision budget to. A 27B doesn't, and I'm not going to pretend it does.
Where a local 27B passed our tests
Now the flip side, and it's a real one. On bounded, in-distribution, testable tasks, the local 27B solved what we set it. In these runs, unattended and checked by running the tests, the local agent built:
| Build | Result (verified by running the tests) |
|---|---|
Bank Account class with overdraft protection | 13/13 tests passing |
| 5-file library management system | Correct borrow/return + edge cases |
| Contacts app, sqlite-backed | Persists and queries correctly |
| Arithmetic expression evaluator | Precedence-correct: 2+3*4=14, (2+3)*4=20 |
| 33-file autonomous build | Every file compiles |
These aren't cherry-picked one-liners; they're multi-file programs with real edge cases, and they pass the tests. Passing these tests is not the same as coding like Claude: Claude Code is still faster and better at editing large codebases.
And there's a quieter property that matters more than any single build: it's truthful. When a build's tests fail, the agent reports the failure honestly instead of declaring a false success. A coding agent that lies about passing is worse than useless; one that tells you the truth about where it stands is one you can actually delegate to.
Where it falls short of Claude
- Breadth-bound tasks: the ones that hinge on knowing an obscure fact or library quirk (wall #2).
- Flawless multi-step novel reasoning with no test to check against: no execution oracle can save you when there's nothing to run.
- 40-plus-step autonomous runs: error compounds over a long horizon; small models drift where Claude holds the thread.
The biggest lever: an execution oracle
If you take one thing from this page, take this. The change that helps most is how you select the answer. There are three ways to pick which of a model's attempts to trust, and they are wildly different in value:
| Selection method | What it actually checks | Trustworthy? |
|---|---|---|
| The model approves itself | Whether the model thinks it's right | No, a confident wrong answer looks identical to a right one |
| "It compiles" | Syntax, not behavior | Weak: compiling code is routinely wrong |
| Execution oracle (run the tests) | Whether the behavior is actually correct | Yes, the ground truth |
Selecting on run-the-tests is what turns a small model's noisy best-of-N attempts into a reliable pass. The model doesn't have to be right on the first try: it has to be right on one try, and the oracle finds it. This is the lever behind most of the local-engineering jump in the roadmap above, and it's the one Outlier leans on hardest.
Self-verification: the agent checks its own work
The newest piece extends the oracle inward. When a test fails, the agent now computes the true answer itself and fixes whichever side is actually wrong: the code, or a test that asserts a wrong value. Crucially, there's a guard that blocks weakening a good test to cheat the check. That guard is the whole game: an agent allowed to edit tests will, if you let it, "fix" a failure by deleting the assertion. Blocking that closes the biggest loophole in self-verification, though not every one: changing an expected value still rests on the agent's own worked answer.
The honest takeaway
Can a local 27B model code like Claude? Not across the board. On bounded, in-distribution, testable work, it passed our tests: measurably, today. Beyond that, there's a real gap with three structural causes I won't hand-wave away. The interesting truth is how much of the gap may be engineering rather than raw capability: selection, memory, and edit strategy don't touch the model at all, though I have not yet measured how many points they are worth. That's the seam a local coding agent lives in: private, offline, free of usage caps, and honest about where it stands. When the task hits a wall, the right move isn't to pretend; it's to escalate. Everywhere else, the local model earns its keep.
Questions about the 27B coding measurements
Can a local 27B model code like Claude?
Not across the board. On bounded, in-distribution, testable coding tasks, a local 27B model built and passed the tests for a bank account class (13/13), a 5-file library system, a sqlite contacts app, and a precedence-correct arithmetic evaluator this session. But a genuine gap remains on breadth-bound work, flawless multi-step novel reasoning with no test to check against, and 40-plus-step autonomous runs.
What did the local 27B score on SWE-bench Verified?
Measured blind, the local 27B agent scored 46%: 23 of 50 tasks resolved. That is a measured number on real tasks, not a leaderboard entry, and not the full 500-task Verified set.
What is the single biggest lever to close the gap?
An execution oracle: selecting the candidate patch by running the tests, not by whether it compiles and not by letting the model approve its own work. Running the tests turns a small model's noisy best-of-N attempts into a reliable pass, and it is the lever that moves the score most.
Try Outlier free
Free: Nano + Lite. Pro: $39 once · Black Friday sale: $16.99 once through December 2. Then $39. macOS 15+. Refund window: 30 days, no questions asked. Priced in US dollars. Checkout shows your currency and any local tax.
Download free (277 MB) Buy ProOn your phone?