How to make a local AI run your tests, not just write code
- Most local-AI tools write code. Far fewer execute it, and that gap is where the real work lives.
- Outlier's Code Agent edits files, runs the tests, reads what failed, and tries again — on your machine.
- This is the loop our SWE-bench harness drives end to end, which is why we can publish a number for it at all.
- It never touches the network to do it: the model, the repo and the test run are all local.
There is a large difference between an assistant that produces a plausible patch and one that finds out whether the patch works. The first hands you a suggestion; the second hands you a result. Running the tests is what separates them.
Why "it writes code" is the low bar
Any competent model can produce code that looks right. The question that matters is whether it compiles, whether the suite still passes, and whether the specific failing test now passes. Answering that requires execution, not generation.
When we checked six local-AI tools' own documentation, several describe editing files. None documented executing a test suite and iterating on the failures. That is the capability gap, and it is the one worth caring about.
The loop
- Point it at a project. The agent indexes the codebase so it can find the right file instead of guessing.
- Describe the change in ordinary words — a bug to fix, a behaviour to add.
- It edits, then runs your tests and reads the output.
- On a failure it iterates rather than declaring victory, and you see the test progress as it goes.
You approve what it is allowed to run. An agent with shell access on your machine should be something you opted into, and it is.
Be realistic about what a local model solves
We measure this against SWE-bench Verified, blind — no oracle, no test patch in the prompt — using the official Docker harness. A 27B model on a laptop solves a real fraction of those tasks, not most of them.
Agentic scores are also harness-dependent: a scheduling fix once moved a tier from 11/30 to 15/30 with no change to the weights. Anyone quoting an agent number without saying which harness produced it is quoting a number they don't understand.
The honest framing: this is a capable junior that never gets tired, never sends your code anywhere, and will happily grind a failing test at 2am.
Where it earns its keep
- Code you cannot upload. Client work, regulated code, anything under NDA.
- Long unattended jobs, where a per-token bill would make you think twice about letting it iterate.
- Flaky-test triage, where the loop is run, read, adjust, run again.
- Offline work — it does not degrade on a plane, because nothing was remote to begin with.
Common questions
Can a local AI actually run my test suite?
Yes. Outlier's Code Agent executes commands on your machine with your approval, so it can run the suite, read the failures and iterate. That execution loop is what our SWE-bench measurements exercise.
Is my code sent to a server?
No. The model runs on your Apple Silicon chip and the repository stays on disk. There is no upload step, which is the reason this is usable on code you are contractually unable to share.
How good is a local coding agent really?
Good enough to be useful, not good enough to be unattended-and-unreviewed. We publish blind SWE-bench numbers rather than adjectives, and we state the harness, because agentic scores move with the harness.
Which tier should I use for coding?
Code 27B. It shares Core's weights with code-specialized configuration and measures 89.5% on MMLU (n=200). Quick is a reasoning tier and is not positioned for agentic coding.
Try it on your own Mac
One signed, notarized download. No account, no token bill, and it keeps working with the Wi-Fi off.
Get Outlier for Mac