When a smaller model is not a worse model, and when it is
- Quantisation shrinks a model by storing its weights at lower precision. Done carefully it costs almost nothing.
- Our 27B at mixed 3/4-bit came to 13.3 GB and streamed on a 16 GB Mac with differences from the 4-bit build inside the noise of our test.
- Pushing further, to a mixed 2/6-bit build, dropped MMLU to 32%. On a four-choice test, guessing scores 25%.
- So the answer is not "smaller is worse". It is that there is a cliff, and it is closer than the file size suggests.
Outlier Pro: $39 once · Black Friday sale: $16.99 once through December 2. Then $39.
Free to start (Nano and Lite). macOS 15+, Apple silicon. Refund window: 30 days, no questions asked. Priced in US dollars. Checkout shows your currency and any local tax.
We wanted our 27B tier to run on a 16 GB Mac. The obvious lever is quantisation, and the obvious question is how far it can be pushed before the model stops being the model.
What held
A mixed 3/4-bit build brought the weights to 13.3 GB, small enough to stream on a 16 GB machine. On our evaluation it was near-lossless: the differences from the shipping 4-bit build were inside the noise of the test. That is a real result, and it is the one we are pursuing.
What broke
A more aggressive mixed 2/6-bit build scored 32% on MMLU. That number is the interesting part, because MMLU is four-choice multiple choice. Random guessing scores about 25%. A model at 32% is not a degraded model. It is barely distinguishable from a coin.
We treat any score near chance as a signal to check the harness before believing the model, because a broken prompt path produces exactly the same number. In this case the harness was fine and the build really had collapsed. Two-bit is dead for this architecture on our evaluation, and we stopped rather than tuning around it.
Why the cliff is sharp
Precision loss does not degrade a model evenly. Most weights tolerate coarse rounding; a small number carry structure that falls apart without fine resolution. Mixed-precision schemes exist to spend the bits where they matter. At 3/4-bit there are enough bits left for the sensitive parts. At 2-bit there are not, and no amount of clever allocation elsewhere compensates.
What this means if you are choosing a local model
File size is a poor proxy for quality across quantisation levels. Two builds of the same model, one 40% smaller, can be equivalent or one of them can be broken, and the download page will not tell you which. Look for an evaluation at the exact quantisation you are downloading, and be suspicious of any score close to what guessing would give.
Method
- Base model: our 27B tier. Builds compared: shipping MLX 4-bit, a mixed 3/4-bit variant at 13.3 GB, and a mixed 2/6-bit variant.
- MMLU, four-choice, deterministic sample, greedy decoding at temperature 0 with a fixed seed. Chance is 25%.
- Measured on an M1 Ultra Mac Studio. The 2/6-bit result was 32%.
- Not yet shipped. The 3/4-bit build is a candidate we are still evaluating, not a tier you can download today.
Outlier publishes what its tiers actually score, including the experiments that failed. The FAQ covers the shipping lineup, and support is one person.
Outlier Pro: $39 once · Black Friday sale: $16.99 once through December 2. Then $39.
Free to start (Nano and Lite). macOS 15+, Apple silicon. Refund window: 30 days, no questions asked. Priced in US dollars. Checkout shows your currency and any local tax.