Outlier › data

Outlier tier benchmarks: MMLU on every shipping tier

Matt Kerr · Outlier · published 2026-08-09 · re-measured 2026-08-24

Every shipping tier's MMLU score, measured through the app people actually download. The raw record is dataset.json (CC‑BY‑4.0).

MMLU, build 1.11.804, 2026-08-24

How these were measured

Mounted the shipped 1.11.804 DMG, asserted /health reported 1.11.804, and queried that sidecar. Every row records the model that produced it and every run recorded zero transport errors.

Deterministic stratified sample across all 57 MMLU subjects (round-robin by subject, no RNG, reproducible). Zero-shot: question + A-D choices, answer with the letter only. Scored by letter extraction; the scorer self-validates against 6 known cases before every run. Requests go through the shipping app's /chat path so the measurement exercises the same code users run.

Decoding: greedy, temperature 0, fixed seed, thinking disabled. Sample: n = 200.

95% interval at n=200 is roughly +/-7 points near 50% and +/-4 points near 90%. Lite and Quick are a statistical tie.

What changed, and why the numbers moved

Re-measured because three of these figures had no surviving results file. nano, lite and quick each came in slightly BELOW the 2026-08-09 run (1.0, 0.5 and 0.5 points). Sampling noise scatters both ways, so a one-sided shift suggests a small systematic difference between builds 1.11.757 and 1.11.804. Every difference is far inside the stated interval. The retired Code tier is dropped: it was Core's weights under a second name and never an independent measurement.

The original run. Kept because the figures above replaced it on the site. Superseded figures are kept rather than deleted, because a number that was published once should stay checkable.

HumanEval, build 1.11.804, 2026-08-24

Full HumanEval, all 164 problems. The model is asked to complete the function; the completion is assembled with the problem's own test suite and check(entry_point) and executed in a subprocess with a 10s timeout. A solution counts only if the official tests pass. The scorer self-validates before any run: a known-correct completion must pass AND a known-wrong one must fail.

These two are tied. Core and Vision 3.8 were given the IDENTICAL 164 problems, so the right test is paired, not a comparison of two independent percentages. McNemar on the discordant pairs: Core solved 4 that 3.8 missed, 3.8 solved 3 that Core missed, 152 both, 5 neither. Exact two-sided p = 1.000. On single-function coding these two models are not distinguishable at this sample size.

An independent cross-check

An earlier independent run at n=50, using a different harness, gave nano 52, quick 82 and core 90 - within one point of these results on every tier.

SWE-bench and decode speed, as shown on the home page

SWE-bench Verified, blind (the agent gets the issue and the repo at base commit, no test patch, no file hint, no pass/fail feedback) graded by the official swebench.harness Docker evaluation: 23 of 50 on a fixed sample (seed 42), seven of which produced no patch at all and are counted as unresolved, ±14 points at this size. An older build scored 18 of 40 on 2026-06-25: statistically the same, though the harness changed a great deal in between. Quick: 0 of 50 on the same sample and protocol (build 1.11.788, 2026-08-17), with no patch produced in any of the 50 runs; an earlier internal set of 30 also scored 0. Decode measured on an M1 Ultra, V9 paged engine; your Mac will differ.

The download is a later build than the one measured; these figures have not been re-run on it.