A benchmark score is usually a model's first guess. But the gap between what a model can do (one of several tries is right) and what it does on the first try is large — and closing that gap is where most of the real capability lives. You don't need a bigger model to get there; you need better systems around the model.
Most published scores are pass@1: is the model's single first attempt correct? But sample the same model several times and one of those attempts is correct far more often — that's pass@k. For local coding models the gap is big, which means the model frequently generates a correct solution but doesn't rank it first. Recover that gap and the same weights get much more capable.
1. The agent loop. Instead of one-shotting, the model localizes the bug, makes a small edit, re-runs the tests, and iterates. This alone is the biggest single jump.
2. Best-of-N + an execution oracle. Generate several diverse attempts, then run the tests to pick the one that actually passes. The oracle is the whole game — without running code you can't tell which attempt is right; with it, you capture pass@k instead of pass@1.
3. Retrieval. For coding, finding the right file/function in the repo matters more than the web. For factual questions it flips — a small model with great retrieval rivals a big one, because that's recall, not reasoning.
4. Escalation. The same oracle that selects also tells you when the local model failed — so you route only that hard tail to a larger model. Local-first, cloud only when it's genuinely needed.
| Setup | Blind SWE-bench (directional) | |
|---|---|---|
| Single-shot (one guess) | ~6% | measured |
| Agent loop, greedy — before we fixed our own harness | 33% | measured |
| Agent loop, greedy — after the July harness fixes | 47% | measured |
| + retry-on-empty (one resample of failed attempts, 1.23× cost) | 53% (n=30) | measured |
| Full stack, firmed on a 3× larger sample | 46% (n=100, 95% CI 37–56%) | measured |
| Contamination-controlled: SWE-bench-Live, tasks created after 2025-03 (the model cannot have seen the fixes), counting only instances whose official human fix passes in our sandbox | 21% (7/33 env-validated, 95% CI ≈ 10–38%; an earlier 9-instance read said 33% — tripling the sample corrected it down, same as our Verified firming) | measured |
| + escalate the hard tail to the cloud | ≈ frontier-cloud level | estimated |
Updated July 24, 2026 — we firmed the number on 100 tasks. The 53% above was measured on 30 tasks; tripling the sample settles the best estimate at 46% (the 30-task figure sits inside the larger sample's confidence interval — small benchmarks flatter). Both rows stay published because showing the sample-size correction is the point of this page. Earlier update: Same 30 blind SWE-bench Verified tasks, same official Docker grading, same on-device 27B model throughout. Greedy first measured 33%. Then we found and fixed a string of bugs in our own harness — a cache that threw away its own work, a four-token prompt mismatch that broke every multi-turn reuse, tool calls our parser silently dropped, duplicate calls a weak guard let through — and the identical model scored 47%. Adding one policy — if an attempt ends with no patch, resample it once at slightly higher temperature (1.23× total inference) — took it to 53%, above the two-sample coverage ceiling we'd previously measured at twice the cost. Two honest walk-backs from earlier versions of this page: the "hotter sampling beats greedy" lead from a 10-task run did not survive n=30 (both 33%; we changed nothing), and the "prefer the smaller diff" tiebreak we called promising did not survive a significance test (it flips on whether you count lines or characters; we shelved it — the drop-empty half stands on logic alone). The recurring lesson we keep re-earning: before concluding a local model can't do something, audit the harness that measured it.
There's a real limit: no selection system can beat the model's pass@k saturation. If many attempts still never contain a correct one, selecting doesn't help — only a better base model or the cloud crosses that. That's exactly why the right architecture is local-first with optional escalation: capture everything the on-device model can do (privately, for free, offline), and reach for a bigger brain only for the tail it genuinely can't reach.
Most of the local-AI space markets model names and vibes. We think the interesting question — and the honest one — is how much of a model's real capability a system actually recovers, measured, with the estimates called estimates and the small samples called small. That rigor is the product: a 10-task run once showed a 70% coverage number and a "hotter sampling wins" lead. We didn't ship either — we ran 30 tasks, the 70% settled to 47% and the sampling lead vanished, and this page now says so. A benchmark you can trust is one whose author tells you when their own earlier number was too rosy.
Download Outlier for MacApple Silicon (M1–M4), macOS 12+. Runs on your Mac; nothing leaves the device unless you enable cloud escalation.
Outlier runs on your Mac. macOS 12+ on Apple Silicon. Back to home.
Open weights ship constantly and most are not worth your disk. When one beats a tier Outlier already has, on hardware you already own, we say so and what it replaces. That is the only reason we email.
Your address and which site you sent it from. No IP, no user agent, no referer. Nothing is sent until you press the button. Privacy.