{ "name": "Outlier tier benchmarks, August 2026", "description": "MMLU, agentic SWE-bench and decode speed for every shipping Outlier tier, measured on an M1 Ultra against the shipping v1.11.757 build. Unmeasured tiers are marked as such rather than estimated.", "url": "https://outlier.host/data/outlier-tier-benchmarks-2026-08/", "creator": "Outlier (Matt Kerr)", "license": "CC-BY-4.0", "datePublished": "2026-08-09", "dateModified": "2026-08-24", "hardware": "Apple M1 Ultra, 64 GB unified memory, macOS", "app_version": "1.11.804", "mmlu": { "method": "Deterministic stratified sample across all 57 MMLU subjects (round-robin by subject, no RNG, reproducible). Zero-shot: question + A-D choices, answer with the letter only. Scored by letter extraction; the scorer self-validates against 6 known cases before every run. Requests go through the shipping app's /chat path so the measurement exercises the same code users run.", "n": 200, "decoding": "greedy, temperature 0, fixed seed, thinking disabled", "date": "2026-08-24", "build": "1.11.804", "measured_through": "Mounted the shipped 1.11.804 DMG, asserted /health reported 1.11.804, and queried that sidecar. Every row records the model that produced it and every run recorded zero transport errors.", "confidence_note": "95% interval at n=200 is roughly +/-7 points near 50% and +/-4 points near 90%. Lite and Quick are a statistical tie.", "results": { "nano": { "accuracy_pct": 52.0, "correct": 104, "n": 200 }, "lite": { "accuracy_pct": 77.0, "correct": 154, "n": 200 }, "quick": { "accuracy_pct": 81.0, "correct": 162, "n": 200 }, "core": { "accuracy_pct": 89.5, "correct": 179, "n": 200, "note": "Reproduced the 2026-08-09 figure exactly (179/200) on a later build, which is why this row is unchanged while the smaller tiers each moved by half a point to a point." }, "vision38": { "accuracy_pct": 82.5, "correct": 165, "n": 200, "note": "Vision 3.8 answers text-only questions, so MMLU is measurable through the product. The superseded record below says Vision could not be measured; that was true of the retired Vision 3.6, which declined text-only requests, and is not true of 3.8." }, "plus": { "measured": false, "reason": "The 209 GB weights are not currently on the test machine (131 GB free), so MMLU has not been run. This is a download gap, not a capability limit: Plus runs on this 64 GB M1 Ultra at 2.1 tok/s with roughly 10 GB resident, streaming experts from SSD via the V9 paged engine." } }, "cross_check": "An earlier independent run at n=50, using a different harness, gave nano 52, quick 82 and core 90 - within one point of these results on every tier.", "change_note": "Re-measured because three of these figures had no surviving results file. nano, lite and quick each came in slightly BELOW the 2026-08-09 run (1.0, 0.5 and 0.5 points). Sampling noise scatters both ways, so a one-sided shift suggests a small systematic difference between builds 1.11.757 and 1.11.804. Every difference is far inside the stated interval. The retired Code tier is dropped: it was Core's weights under a second name and never an independent measurement.", "superseded": { "build": "1.11.757", "date": "2026-08-09", "note": "The original run. Kept because the figures above replaced it on the site.", "results": { "nano": { "accuracy_pct": 53.0, "correct": 106, "n": 200 }, "lite": { "accuracy_pct": 77.5, "correct": 155, "n": 200 }, "quick": { "accuracy_pct": 81.5, "correct": 163, "n": 200 }, "core": { "accuracy_pct": 89.5, "correct": 179, "n": 200 }, "code": { "accuracy_pct": 89.5, "correct": 179, "n": 200, "note": "Identical to Core by construction: Code is Core's weights with a code-specialized configuration. Run as a consistency check, not as an independent measurement." }, "plus": { "measured": false, "reason": "The 209 GB weights are not currently on the test machine (131 GB free), so MMLU has not been run. This is a download gap, not a capability limit: Plus runs on this 64 GB M1 Ultra at 2.1 tok/s with roughly 10 GB resident, streaming experts from SSD via the V9 paged engine." }, "vision": { "measured": false, "reason": "Not measurable through the product. Outlier Vision answers questions about an attached image and declines text-only questions, so a text MMLU score would describe a capability the app does not expose. Verified on 2026-08-09 against the shipping app path: a text request to the vision tier is refused, and the server continued to report the previously loaded text model." } } } }, "swebench": { "method": "SWE-bench Verified, blind: the agent receives the issue text and the repository at base commit only - no test patch, no candidate-file hint, no FAIL_TO_PASS feedback. Patches graded by the official swebench.harness Docker evaluation. Fixed random sample, seed 42. Instances that produce no patch are counted as unresolved.", "caveat": "Agentic scores depend on the agent harness as much as the model - one scheduling fix moved a tier from 11/30 to 15/30 with no change to the weights. Core has now been measured on both an old and the current build; the blind score did not move.", "results": { "core": { "resolved": 23, "n": 50, "resolved_pct": 46.0, "protocol": "blind, official Docker", "date": "2026-08-09", "app_version": "1.11.757", "empty_patches": 7, "confidence_note": "95% interval ~ +/-14 points at n=50", "previous": { "resolved": 18, "n": 40, "resolved_pct": 45.0, "date": "2026-06-25", "note": "older build; retained because the comparison is the point - the harness changed a great deal and the score did not." } }, "quick": { "resolved": 0, "n": 50, "resolved_pct": 0.0, "protocol": "blind, seed 42 - same protocol and sample as Core's 23/50", "date": "2026-08-17", "app_version": "1.11.788", "empty_patches": 50, "note": "All 50 instances produced an empty patch: the agent never emitted a diff, so nothing was submitted to the Docker evaluation. The 0 follows from this method's rule that a no-patch instance counts as unresolved - it is a generation outcome, not a grading one. Quick is strong at chat coding (21/23 vs Code's 20/23 on a 54-prompt set) and this does not contradict that; the agentic loop is the failure, not the model's coding ability.", "confidence_note": "95% interval ~ +/-7 points at n=50; the result is 0 of 50, not a near-zero.", "previous": { "resolved": 0, "n": 30, "resolved_pct": 0.0, "protocol": "internal firm30 agentic set", "date": "2026-07", "note": "Replicated at the larger n and under the blind protocol; same answer." } }, "code": { "measured": false, "reason": "Shares Core's base weights; not separately measured on the blind set." }, "nano": { "measured": false }, "lite": { "measured": false }, "plus": { "measured": false }, "vision": { "measured": false } }, "date": "2026-08-09", "n": 50, "build": "1.11.757", "confidence_note": "95% interval at n=50 is roughly \u00b114 points. The prior measurement, 18/40 = 45.0% on a 2026-06-25 build, is statistically the same result.", "denominator_note": "Scored strictly on the seed-42 n=50 sample. The run artifact held 54 rows; four were leftovers from an earlier run and are excluded. Seven of the fifty produced no patch and are counted as unresolved." }, "decode_tokens_per_sec": { "method": "M1 Ultra, V9 paged engine. Speed varies by Mac.", "source": "https://outlier.host/data/local-ai-benchmarks-mac-2026/", "results": { "nano": 71.7, "lite": 53.4, "core": 20.7, "code": 20.7, "plus": 1.59, "vision": 16.31 }, "source_dataset_last_updated": "2026-05-20", "note": "Carried over from the local-AI-benchmarks dataset (last updated 2026-05-20), so these are an older measurement round than the MMLU and SWE-bench columns, which were taken on 2026-08-09. Speed varies by Mac. Plus is the exception: 1.59 tok/s comes from the pre-ship K-override sweep at the shipped K=20 setting (5 prompts x 80 tokens, isolated subprocess, model load excluded and timed separately at 3.0 s). Per-prompt rates in that run were 1.35/1.52/1.62/1.65/1.64. The engine-comparison dataset separately reports ~2.1 tok/s for V9 paged under its own per-token RSS protocol; where the two disagree this dataset publishes the lower, shipping-config figure." }, "note": "Published because a benchmark table without its conditions is not a benchmark. Every figure states its sample size, decoding settings and date; every gap says why it is a gap.", "humaneval": { "method": "Full HumanEval, all 164 problems. The model is asked to complete the function; the completion is assembled with the problem's own test suite and check(entry_point) and executed in a subprocess with a 10s timeout. A solution counts only if the official tests pass. The scorer self-validates before any run: a known-correct completion must pass AND a known-wrong one must fail.", "n": 164, "decoding": "greedy, temperature 0, thinking disabled", "date": "2026-08-24", "build": "1.11.804", "results": { "core": { "pass_at_1_pct": 95.1, "passed": 156, "n": 164 }, "vision38": { "pass_at_1_pct": 94.5, "passed": 155, "n": 164 } }, "paired_comparison": "Core and Vision 3.8 were given the IDENTICAL 164 problems, so the right test is paired, not a comparison of two independent percentages. McNemar on the discordant pairs: Core solved 4 that 3.8 missed, 3.8 solved 3 that Core missed, 152 both, 5 neither. Exact two-sided p = 1.000. On single-function coding these two models are not distinguishable at this sample size.", "contamination_caveat": "HumanEval was published in 2021 and is very likely inside the training data of any 2026 model, so 95% partly measures memorisation rather than coding ability. We publish it because it is comparable across the tiers here - both faced the same contaminated set - not because it predicts how a model handles your codebase. The agentic SWE-bench figures are the harder and more honest test." } }