The most powerful AI
that never leaves your Mac.

Dot size follows disk footprint. Plus is absent because it has no MMLU run on the shipping 4-bit build — the figure this chart plots. Vision 3.8 has one, and it is in the table: it shares Core’s 24 GB bracket, so on an axis of required memory it would sit on top of Core rather than tell you anything new.

Decode speed on this M1 Ultra, log scale — the tiers span 45×, so a linear axis would flatten Plus to nothing. These figures come from an earlier measurement round (2026-05-20) than the MMLU and SWE-bench numbers above, which were taken 2026-08-09; the dataset file says so too. Quick is blank for the same reason Plus is absent from the chart above: no run on the shipping build. Vision 3.8 is measured but shares Core’s memory bracket, so it appears in the table rather than the chart. Speed varies by Mac.

How these were run — HumanEval: all 164 problems, greedy (temperature 0), execution-graded against the official test suite on the same v1.11.804 build; only Core and Vision 3.8 have been run. Read it as a comparison BETWEEN those two tiers, not as a headline — HumanEval dates from 2021 and is almost certainly inside any 2026 model’s training data, so the absolute number partly measures memorisation. What it does show is that the two 24 GB tiers are level on single-function coding, 156/164 against 155/164, a tie on a paired test, while MMLU separates them by 7 points. How these were run — MMLU: n=200, deterministic stratified sample across 57 subjects, greedy (temperature 0, fixed seed), thinking off, on an M1 Ultra. All five were re-run on 2026-08-24 through the shipping v1.11.804 path — so every figure here comes from one regime, on one build. The page now links 1.11.853. Between the benchmark build and it, 1.11.822 removed a retired tier from the first-run picker, fixed a dialog that promised a fallback and then did nothing, and changed how the code agent explains one refusal: asking it for a shell loop used to return "binary not in allowlist: 'while'", naming a program that does not exist. 1.11.823 then allowed the code agent to run strings and tr: a bench run on the previous build spent 3 of its 17 tool calls being refused on exactly those two while inspecting a binary, which is the one job strings exists for. 1.11.824 then fixed a defect that had stopped the code agent adding a function to an existing file at all: the description of its own edit tool demonstrated a call without the required file path, so every attempt was rejected and the agent reported that it could not do it. Measured on the same task, 0 of 4 runs succeeded before the fix and 5 of 5 after, verified against the built app rather than the source. It does not yet reliably commit that edit — the agent sometimes makes the change, says it is committing, and stops; that defect is open and the edit is usually already on disk. 1.11.825 then stopped the agent damaging a file it edited: when the model's tool call was slightly malformed — a dropped closing tag, which small models do routinely — the recovery that salvages the call could carry that tag, and the chat template's end-of-turn marker, into the contents it wrote, leaving a line like return a - b</args></tool> on disk and a file that no longer parses. Recovering a malformed call is still right; carrying the envelope into the file was not. Unlike the two figures above, this one is not a measured rate — the damage was intermittent, and what changed is that every string a tool receives is now cut at the first such marker, the file path included. 1.11.826 changed how the app updates rather than how it codes: an update that had downloaded used to wait for the next time you opened Outlier, so a copy left running never picked it up — it now restarts itself once you are not mid-task, and writes every check to ~/.outlier/updater.log. 1.11.831 then fixed how it decides you are not using it: 826 waited 20 seconds after your last keystroke, which is what reading a long reply looks like, so it now waits until the window has been in the background for five minutes or the machine itself has gone untouched, never while a reply is on screen in front of you — and it puts your window back where it was, which nothing had ever done. 1.11.832 then fixed two places where a secret could be written down: a bug report or crash bundle included a password or API key when it was stored under a field with that name, because the scrub read the value on its own and the word that made it recognisable stayed behind on the key — and the local activity log kept a key passed to a connected tool, because that scrub stopped at the first container and tool arguments arrive as one. The same log names the project each run belonged to again; every entry had been reading as the same redacted string. 1.11.833 then fixed a limit that stopped the largest model starting at all: Vision 3.8 was given the same five-minute allowance as the smallest tier, so on a real project it was cut off before producing a single word — and the app reported success while doing nothing. The allowance is now worked out from the model’s own size rather than a list someone has to remember to update. 1.11.834 then removed a price the checkout had stopped charging: a $9/month plan, unsold since payments moved providers, was still quoted inside the app beside a website that says there is no subscription. The figure now comes from the same record the checkout is paid from, so the two cannot drift apart again. 1.11.835 then closed a way to lose one of the three machines a licence covers: entering the key a second time on the same Mac gave the new activation a new id and left the old one holding a seat that nothing could hand back. The previous activation is now released before the new one is stored, and if the licensing service cannot be reached at that moment the details are kept and retried rather than lost. 1.11.836 then stopped Outlier reporting a machine as released when it had not been: the licensing service answers a release with an empty acknowledgement, which was read as success, so a release that quietly failed left the machine counted against you with nothing recording it. Outlier now checks afterwards that the machine really is released, and keeps the details to retry when it cannot check. 1.11.837 then gave Outlier a little more room to finish: on a long task it now knows how much of its own working budget is left, and if it has already started changing your files and is running low it gets a few more steps, once, to finish the job and check its work rather than stopping partway. 1.11.838 then finished that job on the other of the two licensing services, and stopped a question costing you a machine: releasing a Mac is now confirmed with whichever service actually issued your key rather than assumed from an acknowledgement, and a release that cannot be confirmed keeps the details and retries instead of discarding the one thing that can hand the seat back. Asking the app whether you are licensed used to be able to spend a machine, because the check that answered the question could also activate; it now only reads. And an update check that lands while a new version is still uploading waits and asks again a few minutes later rather than telling you that you are up to date. 1.11.839 then fixed a step that looked like it had worked when it had not: connecting a folder to a project in one go returned you a project with nothing attached, and you only found out later when the app said the project had no codebase. Attaching a folder has always needed its own step, because that step is what reads the files — skipping it would leave a project claiming a codebase nobody had looked at. That was a good reason and the app had never said it out loud; now it does, and it points at the thing to do instead. The same release gave the local API back its own documentation: twenty-three endpoints had quietly stopped describing themselves eighteen releases ago, so anyone reading the built-in API reference saw blanks where the explanations should be. 1.11.840 then fixed an update that could leave Outlier unable to start at all. When the app updated itself it restarted without shutting its engine down first, so the old engine kept the port the new one needed and the new copy came up with nothing to talk to. A safety net exists for exactly that leftover process, and it had stopped recognising it eighty-eight days earlier — it was matching a shortened form of the name that this system has never actually printed, so on every launch it found the process, decided it was not ours and left it, while writing a tidy line to the log that reads like a decision rather than a fault. That is why it lasted 290 releases. Both halves are fixed, and because the safety net runs before the engine is started, installing this build clears a Mac that is already stuck: there is nothing to run by hand. 1.11.845 then told the agent that it is allowed to use your Mac. Asked to build and launch something — a game, a small app — it would write the code and then say it was unable to run it, or ask you to open a terminal and do it yourself. Nothing was broken: it had simply never been told, in the instructions it actually reads, that it can run commands and open applications on the machine it is already running on. Now that is the first thing it is told, before it is shown a single tool. Two habits that made it give up early went with it. When the model’s request to use a tool came out slightly malformed — a dropped closing tag, which smaller models do routinely — the retry meant to repair it also offered a way out, answer in plain text instead, and the smaller tiers took that way out nearly every time: the code was printed into the chat, your file was never touched, and the work was reported as done. That escape is gone, and a turn that ends without doing what was asked now says so instead of reporting success. Measured on the same request on 1.11.840 — add a function to an existing file — the smallest tier finished 3 of 6 runs, while Core and Vision 3.8 finished every one; starting an agent run on a tier measured unreliable at using tools now suggests a larger one first, which the chat side already did. The same release restored the Preview button on something the app had just written: a page it produced without a code fence around it was shown as text rather than offered to run, so a working drawing app looked like a wall of HTML. 1.11.846 then fixed an app the model never finished being handed to you as if it were finished. Ask Outlier to build something and the model sometimes stops part-way through the file — after the styles and half the toolbar, before the drawing area and the code that makes it work. Two recoveries existed for exactly that, and both had been switched on for months: one resumes writing from where it stopped, the other rewrites the file properly. Neither had ever run. Before either could look at the answer, Outlier had already added the closing tag itself, so that a half-written page would at least render — and both recoveries decide whether to step in by asking whether the document looks unfinished. By then it no longer did. Outlier closed the file, then asked itself whether the file was closed, and believed its own answer. Measured on 1.11.845, asking Core for a drawing app: the model stopped mid-way through a toolbar button, about 60% of the way into the file, with no drawing area and no code at all — and asking again in the same conversation for the missing parts did not help, because the recovery had to happen while the model was still writing, which is precisely what the closing tag prevented. The tag is now added once, at the end, after the recoveries have had their turn: you still never see a dangling tag, and an unfinished app gets finished instead of arriving dead. 1.11.847 then fixed the Pro agent, which had received none of the week’s agent fixes. Asked to build a game and launch it, it wrote the whole program into the chat, told you to paste it into a file on your desktop, and reported itself finished; asked to “launch it”, it opened a browser to Apple’s homepage — nothing had been written — and said it had launched the browser. The three fixes that taught the agent it can use your Mac had reached only the standard agent path; the Pro agent builds its instructions separately and never saw them. Both now read the same words from one place, so they cannot drift apart again; when the model writes the finished app instead of a plan, Outlier writes that app into your project and launches it rather than discarding it as text; “launch it” now means the thing you asked for, written first if it does not yet exist, never a browser or a Terminal as a stand-in; and a build that produced no files says so instead of reading as finished. The same release added a gate that drives every agent path on the installed app with the smallest model before a version can be replaced, because these bugs did not fall through the tests — they fell through the configuration the tests ran in. The same release stopped a chat switch from cancelling the reply you were waiting for: it keeps going in the chat it started in, that chat shows a small pulsing dot in the sidebar, coming back shows it still streaming, and only Stop stops it. And you can now reply to just the part of a reply you mean: select a passage and a small “Reply to this” appears beside it; the passage rides at the top of your message as a quote, so the model knows exactly which part you are answering. 1.11.848 then made the model list follow the Mac it runs on. The picker offers Outlier’s own tiers only, marks the one this Mac should run — worked out from its memory, with room left for the system — and a Mac that has never chosen starts on it; a scan on the Models page says which tier is best for this hardware and why, so a bigger Mac gets a bigger model without anyone editing a list, and the list the app shows before the engine answers no longer carries its own copy of the memory figures, which had drifted from the real ones. The same release repaired the crash notice: since 1.11.784 it was meant to blame memory only after measuring it, but the measurement read a table that did not exist, so every crash was reported as “we couldn’t tell why”; it now reads the model’s real size and names memory only when memory is short. And the build itself now refuses to seal an app that is missing any of the three things it must carry — the engine, the bundled Node runtime and the on-screen text reader — where before a missing one would have shipped silently. 1.11.849 then closed three gaps found by driving the installed app through the same task on the smallest model. A program that crashed within seconds of launching was still described as running, because the launch step returned before the program had had time to fail; the reply is now checked against the program’s real exit code, the agent gets one turn to repair the error with the crash log in hand, and if it cannot, you are told it is not running. The wrap-up no longer invents a process id or tells you to launch what it already launched, stray fragments of the agent’s own tool markup no longer reach the screen, and a script the agent wrote runs as ./script.sh instead of being refused. 1.11.850 then closed the next thing the same test found: one run had written two empty files, launched the empty one, watched it end at once and still reported the game running. Writing an empty file is now refused, so the code has to go into the file rather than the reply; a program that ends the moment it starts is reported as finished, never as running; and the pre-release check counts a file only when it has real contents. 1.11.851 then declared the floor the engine had all along — macOS 26 or later — so an older Mac is refused at install instead of after the money, and taught the agent that this Mac has pip3 but no bare pip, a launch that had died on it now running as python3 -m pip. 1.11.852 then stopped the agent fighting a Tk that Apple’s own Python cannot open on macOS 26 — it is told what happened and builds a browser or pygame game instead — and cleared the last stray fragment of its own markup from the start of a reply. 1.11.853 then stopped the coding agent reading a large slice of a project before its first step — it gets the file list, the project’s AGENTS.md or CLAUDE.md, and a few excerpts read fresh from disk — raised its step budget from 20 to 150, told a malformed step exactly what broke, and stopped saving quiz questions it saw in testing as memories about you. None of that reaches the MMLU or HumanEval figures: those are single-shot and greedy, with no agent in the loop. The SWE-bench figure is the one to watch, and not only because it is an agent run — in our own bench the agent read that misleading refusal and recovered by guessing, so a message it can act on is exactly the kind of change that could move an agent score. It has not been re-run since v1.11.804. The SWE-bench figure is a different case and we should say so plainly — it is an agent run, and the last two items in that list are changes to the agent, so that number could move on a re-run. It has not been re-run since v1.11.804, and this sentence will keep saying so until it has. They previously came from two: Nano, Lite, Quick and Core on v1.11.757, Vision 3.8 on a later build. Core and Vision 3.8 reproduced their earlier scores question for question; Nano, Lite and Quick each moved by one or two questions, which is what the re-run was for. At n=200 the 95% interval is roughly ±7 points near 50% and ±4 near 90%, so Lite and Quick are a statistical tie. SWE-bench Verified, blind — the agent gets the issue and the repo at base commit, no test patch, no file hint, no pass/fail feedback — graded by the official swebench.harness Docker evaluation: 23 of 50 on a fixed sample (seed 42), seven of which produced no patch at all and are counted as unresolved, ±14 points at this size. An older build scored 18 of 40 on 2026-06-25 — statistically the same, though the harness changed a great deal in between. Quick: 0 of 30 on our internal firm30 set. Decode measured on an M1 Ultra, V9 paged engine; your Mac will differ.

Why this table is unusual — checked on their live pages 2026-08-13, Ollama, LM Studio, Jan and GPT4All publish no MMLU, HumanEval or SWE-bench figure at all. That is not a criticism — the scores belong to the upstream models — but it means nobody in this category tells you what a local model will actually get right. These are ours to be held to.

Apple Silicon only (M1 / M2 / M3 / M4). Intel Macs are not supported.

Choosing the machine first? M1 vs M4 compares memory bandwidth, Mac mini vs Mac Studio compares 48 GB against 96 GB, and Mac vs PC covers why unified memory decides the model size.

14-day money-back guarantee — just email matt@outlier.host.