When a smaller model is not a worse model, and when it is
- Quantisation shrinks a model by storing its weights at lower precision. Done carefully it costs almost nothing.
- Our 27B at mixed 3/4-bit came to 13.3 GB and streamed on a 16 GB Mac with no quality loss we could measure.
- Pushing further, to a mixed 2/6-bit build, dropped MMLU to 32%. On a four-choice test, guessing scores 25%.
- So the answer is not "smaller is worse". It is that there is a cliff, and it is closer than the file size suggests.
We wanted our 27B tier to run on a 16 GB Mac. The obvious lever is quantisation, and the obvious question is how far it can be pushed before the model stops being the model.
What held
A mixed 3/4-bit build brought the weights to 13.3 GB, small enough to stream on a 16 GB machine. On our evaluation it was near-lossless: the differences from the shipping 4-bit build were inside the noise of the test. That is a real result, and it is the one we are pursuing.
What broke
A more aggressive mixed 2/6-bit build scored 32% on MMLU. That number is the interesting part, because MMLU is four-choice multiple choice. Random guessing scores about 25%. A model at 32% is not a degraded model. It is barely distinguishable from a coin.
We treat any score near chance as a signal to check the harness before believing the model, because a broken prompt path produces exactly the same number. In this case the harness was fine and the build really had collapsed. Two-bit is dead for this architecture on our evaluation, and we stopped rather than tuning around it.
Why the cliff is sharp
Precision loss does not degrade a model evenly. Most weights tolerate coarse rounding; a small number carry structure that falls apart without fine resolution. Mixed-precision schemes exist to spend the bits where they matter. At 3/4-bit there are enough bits left for the sensitive parts. At 2-bit there are not, and no amount of clever allocation elsewhere compensates.
What this means if you are choosing a local model
File size is a poor proxy for quality across quantisation levels. Two builds of the same model, one 40% smaller, can be equivalent or one of them can be broken, and the download page will not tell you which. Look for an evaluation at the exact quantisation you are downloading, and be suspicious of any score close to what guessing would give.
Method
- Base model: our 27B tier. Builds compared: shipping MLX 4-bit, a mixed 3/4-bit variant at 13.3 GB, and a mixed 2/6-bit variant.
- MMLU, four-choice, deterministic sample, greedy decoding at temperature 0 with a fixed seed. Chance is 25%.
- Measured on an M1 Ultra Mac Studio. The 2/6-bit result was 32%.
- Not yet shipped. The 3/4-bit build is a candidate we are still evaluating, not a tier you can download today.
Outlier publishes what its tiers actually score, including the experiments that failed. The FAQ covers the shipping lineup, and support is one person.