How to compare AI models side by side on your Mac

The honest answer to "which model should I use?" is that it depends on what you're asking it. The dishonest answer is a single leaderboard number. Comparing on your own prompts settles it in about a minute.

Why a leaderboard can't answer this

A benchmark score is an average over someone else's task distribution. Your work is not that distribution. We publish MMLU per tier — Nano 52.0%, Lite 77.0%, Quick 81.0%, Vision 3.8 82.5%, Core 89.5%, all n=200 — and those numbers are real, but they cannot tell you whether Lite is good enough for the summarizing you do all day.

Two of our own tiers sit within the confidence interval of each other on MMLU while feeling quite different in use. The only way through that is to try both on something you actually care about.

Where the leaderboard runs out, in our own numbers

Here is the lineup with the gap to the next tier down, because the gap is the part a single column hides. At n=200 the 95% interval is roughly ±7 points near 50% and ±4 near 90% — so a gap smaller than the interval is not a ranking, it is noise wearing a decimal point.

Read the fourth column rather than the second. Three of the tiers measured here are not separated from their neighbour by more than the measurement can see, and one gap — Nano to Lite — is so large that no comparison is needed to feel it. That is the whole case for comparing on your own prompts: for the tiers in the middle, the leaderboard genuinely does not know which is better for you, and it is not being coy about it.

Plus 397B is not in the table, and that is a fact about the measurement rather than the tier: the n=200 MMLU re-run covered five of the six shipping tiers and Plus was not one of them. A table that quietly dropped the sixth would be the kind of thing this page is arguing against.

Comparing

1. Open compare mode from the header.
2. Pick the tiers you're deciding between.
3. Ask once. The prompt goes to each of them.
4. Read them side by side and notice which one you'd actually ship.

Use a real task, not a riddle. The prompts that separate models in practice are the boring ones you repeat daily.

What you'll usually discover

• Smaller is often enough. For summarizing, rewriting and routine questions, a small tier frequently matches a large one — at a fraction of the RAM.
• Size shows up on reasoning. Multi-step problems are where the bigger tiers separate.
• Speed is part of quality. A tier you wait 90 seconds for is worse in practice than a quick one you actually use — which is why we publish Plus at its measured 1.59 tok/s rather than burying it.

Common questions