4-bit vs 8-bit quantization for local LLMs: which should you run?

Quantization isn't a product or anyone's competitor. It's the compression step that makes a model fit on hardware you already own, and every on-device Mac LLM app uses it. The real question is which bit width an app ships, and whether it tells you. Disclosure: I'm Matt Kerr, I build Outlier, a Mac app that ships 4-bit-class models — so check the sources, not me.

What "4-bit" means, and why it isn't 4 bits

Weights are normally 16-bit floats. Quantization rounds them onto a coarse grid: split each tensor into small blocks, find a scale (and for some formats a minimum) for each block's range, then store every weight as a small integer index. The scales stay at higher precision — which is where the naming goes fuzzy.

llama.cpp publishes the real figures for Llama-3.1-8B: Q4_K_M is 4.8944 bits per weight (4.58 GiB), Q8_0 is 8.5008 bpw (7.95 GiB), F16 is 16.0005 bpw (14.96 GiB). "Half a byte per parameter" understates the real footprint by roughly 20%, so size a model against your RAM using the published file size, not arithmetic.

Not all 4-bit is the same 4-bit. K-quants spend more bits on the most sensitive tensors, GPTQ minimizes per-layer output error, AWQ protects salient channels using activation statistics, and QLoRA's NF4 is described as information-theoretically optimal for normally distributed weights. On Apple silicon, MLX supports 2, 3, 4 (the default), 5, 6 and 8 bits, plus fixed-width mxfp4 and nvfp4.

4-bit is faster than 8-bit, not just smaller

This is the part people get backwards. Generation is memory-bandwidth-bound — every token streams the whole weight set through the compute units, so fewer bits means fewer bytes moved. llama.cpp's Llama-3.1-8B run, generation at 128 tokens: Q4_K_M 71.93 t/s, Q6_K 58.67 t/s, Q8_0 50.93 t/s, F16 29.17 t/s. Same hardware, same weights — a 1.4x edge over 8-bit, 2.5x over fp16.

Prefill runs the other way, because prompt processing is compute-bound: F16 923.49 t/s at 512 tokens, Q8_0 865.09, Q4_K_M 821.81. Dequantizing costs you when the ALUs are saturated. So 4-bit buys generation speed, not prompt-ingest speed — for long-prompt, short-answer work the advantage shrinks or inverts.

What you pay for it in quality

The canonical numbers are llama.cpp's own quality table (LLaMA-7B), given as perplexity deltas against fp16: Q4_K_S +0.1149, Q4_K_M +0.0535, Q5_K_M +0.0142, Q6_K +0.0044, Q8_0 +0.0004. Q4_K_M, Q5_K_S and Q5_K_M are labeled "recommended." Q8_0 is marked not recommended — not because it's bad, but because it's overkill: Q6_K is already within 0.005 perplexity of fp16 at 1.5 GB less.

Task scores agree. A 2026 benchmark across 13 llama.cpp quant formats on Llama-3.1-8B-Instruct puts Q4_K_S at an average of 69.17 against Q8_0's 69.41, GSM8K 77.33 vs 77.48 — inside noise, at a 70.83% size reduction instead of 46.87%. Caveat honestly: single-author preprint, not peer reviewed.

8-bit's case is well evidenced too. LLM.int8() reports Int8 inference without performance degradation up to 175B parameters, routing emergent outlier dimensions through a 16-bit matmul while over 99.9% of compute stays in 8-bit. And mlx-lm's docs concede that distilling 16-bit to 8-bit or even 6-bit "often doesn't work well" — the loss starts so low there's nothing left to recover.

Caveat: sizes and throughput come from the Llama-3.1-8B run, the perplexity deltas from an older LLaMA-7B table. Q6_K's 8B size isn't published; on 7B it's 5.15 GB against F16's 13.0 GB.

Where 4-bit genuinely wins

No hedging: the evidence is one-sided, and it's the vendors' own data. Q4_K_M costs about +0.05 perplexity, takes a 13 GB 7B model to under 4 GB, and llama.cpp marks it recommended while calling 8-bit overkill. Ollama ships it as the silent default: llama3.1:latest and llama3.1:8b resolve to the same 4.9 GB artifact as llama3.1:8b-instruct-q4_K_M, identical digest. On Mac, mlx_lm.convert -q gives you 4-bit.

The bigger win is that 4-bit changes which models you can run at all. On a 16 GB Mac it's the difference between a 20B-class model and nothing: gpt-oss-20b was post-trained with MXFP4 quantization of its MoE weights and is documented to run within 16 GB. That family has no higher-precision original, so the choice isn't even offered.

Modern 4-bit isn't the naive 4-bit that gave the technique its 2023 reputation — k-quants, GPTQ, AWQ and MLX's teacher-distilled quants each attack the gap differently. And it's free: llama.cpp and MLX are both MIT-licensed, so the bit width you want is minutes of local compute away.

Where the reassuring numbers don't reach

Perplexity and multiple-choice benchmarks are the metrics least sensitive to the failures users notice. A 0.24-point average-score gap is compatible with a model that emits a plausible but nonexistent API name in generated code, and nobody has published a good measurement of 4-bit's effect on exact-symbol recall.

Damage isn't constant across models, either — it grows with how heavily a model was pretrained. "Scaling Laws for Precision" (Kumar et al., 2024, fit on 465+ pretraining runs) finds post-training quantization degradation rises with training data, to the point where extra pretraining turns actively harmful for the quantized model. An empirical LLaMA3 study across 10 PTQ and LoRA-FT methods at 1–8 bits agreed, for a 15T-token model. Friendly numbers from 2023-era 7B models don't transfer automatically.

4-bit also sits close to a cliff: llama.cpp calls Q3_K_S "very high quality loss" (+0.5505) and Q2_K "not recommended" (+0.8698), so a vendor quietly shipping Q3 to save RAM is selling a materially different product. The published comparisons are short-context, and KV-cache quantization is a separate axis whose long-context behavior is community-reported rather than studied — treat it as unverified. Cross-app comparison is nearly hopeless too: Q4_K_M, NF4, AWQ-int4, MLX affine 4-bit and MXFP4 are all called "4-bit," and none are equivalent.

One shipped example: what Outlier does

Outlier is one instance of the 4-bit default, not the point of this article. It's a Mac app, Apple Silicon only, on-device, no account. You can back the bit width out of the downloads: Nano is 4B in 2.37 GB, Core 27B in 15.13 GB — both near 4.5 bits per weight. Each tier ships as one artifact, so picking a tier is the quantization choice; that's less control than llama.cpp or Ollama give you.

What it buys, measured: Core 27B matched Claude Opus on 98.9% of rubric checks in a 54-prompt head-to-head, and scored about 45% on a blind slice of SWE-bench Verified. Outlier has no web search, image generation, mobile app or Windows build.

Who should pick which

Pick 4-bit for normal work — chat, drafting, coding help, summarization — or whenever it makes a larger model fit your RAM. It's the default for good reason, and it generates faster.

Pick 5-bit or 6-bit if you have headroom. Q6_K measures +0.0044 perplexity, indistinguishable from fp16 on that metric, and still generates faster than 8-bit.

Pick 8-bit when you need quantization removed as a variable: evals, debugging whether a failure is the model or the quant, or work where exact recall beats speed. Cheap insurance if the RAM is idle.

Pick fp16 for prefill-dominated work, or as the reference when you're producing quantizations yourself.

The binary is a bit fake anyway. mlx-lm supports dynamic quantization that estimates per-layer sensitivity, and it's common to keep embeddings and the output projection higher. The honest answer is often "4-bit, with 6-bit in three places."