A benchmark score is usually a model's first guess. But the gap between what a model can do (one of several tries is right) and what it does on the first try is large — and closing that gap is where most of the real capability lives. You don't need a bigger model to get there; you need better systems around the model.
Most published scores are pass@1: is the model's single first attempt correct? But sample the same model several times and one of those attempts is correct far more often — that's pass@k. For local coding models the gap is big, which means the model frequently generates a correct solution but doesn't rank it first. Recover that gap and the same weights get much more capable.
1. The agent loop. Instead of one-shotting, the model localizes the bug, makes a small edit, re-runs the tests, and iterates. This alone is the biggest single jump.
2. Best-of-N + an execution oracle. Generate several diverse attempts, then run the tests to pick the one that actually passes. The oracle is the whole game — without running code you can't tell which attempt is right; with it, you capture pass@k instead of pass@1.
3. Retrieval. For coding, finding the right file/function in the repo matters more than the web. For factual questions it flips — a small model with great retrieval rivals a big one, because that's recall, not reasoning.
4. Escalation. The same oracle that selects also tells you when the local model failed — so you route only that hard tail to a larger model. Local-first, cloud only when it's genuinely needed.
| Setup | Blind SWE-bench (directional) | |
|---|---|---|
| Single-shot (one guess) | ~6% | measured |
| Agent loop (localize → edit → re-test) | ~45% | measured |
| + best-of-N + execution oracle + retrieval | ~60–65% local | estimated |
| + escalate the hard tail to the cloud | ≈ frontier-cloud level | estimated |
The two measured rows are real (our machine, an on-device 27B coding tier, official Docker grading). The two estimated rows are projections — we're running the measurement now and will replace them with real numbers.
There's a real limit: no selection system can beat the model's pass@k saturation. If many attempts still never contain a correct one, selecting doesn't help — only a better base model or the cloud crosses that. That's exactly why the right architecture is local-first with optional escalation: capture everything the on-device model can do (privately, for free, offline), and reach for a bigger brain only for the tail it genuinely can't reach.
Most of the local-AI space markets model names and vibes. We think the interesting question — and the honest one — is how much of a model's real capability a system actually recovers, measured, with the estimates called estimates. That rigor is the product. When the best-of-N number lands, it'll show up here as measured, whatever it says.
Download Outlier for MacApple Silicon (M1–M4), macOS 12+. Runs on your Mac; nothing leaves the device unless you enable cloud escalation.
Outlier runs on your Mac. macOS 12+ on Apple Silicon. Back to home.