GGUF vs MLX: how the two local model formats actually differ

Disclosure first: I build Outlier, a Mac app that runs models on-device, so I have a stake in how people think about local inference. That's why I'd rather explain the mechanism than sell a conclusion. Most GGUF-vs-MLX arguments go wrong in the first sentence, because the two things being compared aren't the same kind of thing.

GGUF is a container, not an engine

GGUF's own spec calls it a file format for storing models for inference with GGML and GGML-based executors. Georgi Gerganov created it — he also wrote llama.cpp — and it replaced the earlier GGML, GGMF and GGJT formats. The stated design goals: single-file deployment, extensibility, mmap compatibility, ease of use, and full information, meaning everything needed to load a model lives inside the file. No external config or loader script.

That last property matters most in practice. Hugging Face draws the contrast directly: unlike tensor-only formats such as safetensors, GGUF encodes both the tensors and a standardized set of metadata. A key-value header carries architecture, tokenizer and hyperparameters, so an old loader can open a new file and skip fields it doesn't recognize. The spec is on v3 — v2 widened count fields to uint64, v3 added big-endian support.

The type table is where GGUF's reach shows: roughly forty tensor and quantization type codes, from F32, F16 and BF16 down through the Q*_K k-quants, the IQ* importance-matrix quants, ternary TQ1_0/TQ2_0, and MXFP4. llama.cpp advertises 1.5-bit through 8-bit integer quantization and ships an imatrix tool that computes an importance matrix over a text dataset to improve quantized quality. The repo makes no quantified claim about how much it helps; neither will I.

MLX is a runtime, not a file format

MLX is an array framework for machine learning on Apple silicon, from Apple's own machine learning research group. It's NumPy-shaped rather than model-shaped: arrays, lazy computation, dynamic graphs, and Python, C++, C and Swift APIs. The defining choice is that arrays live in shared memory, so operations run on CPU or GPU with no transfer between them. Where both already share one pool of RAM, that removes a copy other stacks pay for.

MLX LM is the language-model package on top. Weights are safetensors plus a config, not a bespoke container. You get mlx_lm.generate, mlx_lm.chat, mlx_lm.convert (with -q for quantized conversion), mlx_lm.cache_prompt, LoRA and full fine-tuning, and distributed work through mx.distributed. There's an OpenAI-style HTTP server too, though its own docs say it isn't recommended for production because it only implements basic security checks.

The hardware floor is real: Apple silicon, macOS 14.0 or later, native Python 3.10+. Intel Macs are out. CUDA-on-Linux and CPU-only Linux builds now exist, so "MLX is Apple-only" isn't strictly true, but the Mac path is what everything is tuned for.

Side by side

Where each one genuinely wins

GGUF wins on reach, and it isn't close. 192,308 models carry the gguf library tag on Hugging Face against 19,656 for mlx — roughly ten to one. When a new open model drops, the community usually has GGUF builds at half a dozen bit-widths within hours, and the same file runs on your M-series laptop, your NVIDIA box, an AMD card, or plain CPU. One artifact, every machine you own. GGUF also gets first-party tooling with no MLX equivalent: an inline metadata viewer on model pages, the gguf-my-repo conversion Space, and a JavaScript parser that reads a remote GGUF's metadata without downloading it.

MLX wins on hardware depth and on training. Unified-memory arrays with no CPU/GPU copy is the design you get when the framework authors work at the company that made the chip. MLX also does what a pure inference format can't: LoRA and full fine-tuning on your own Mac, distributed training over mx.distributed, and a Python API that makes a model something you modify, not just run. Its learned-quantization suite is unusually sophisticated for a local stack — DWQ, AWQ, GPTQ and per-layer dynamic quantization, cascadable, aimed at 2-to-4-bit where small-RAM Macs live. The docs are candid: DWQ takes longer than dynamic quantization but typically produces better results, and distilling to 6- or 8-bit often doesn't work well. MLX Swift is the sane route for native Mac or iOS apps.

They're not fully walled off. mlx.core.load reads .gguf alongside .npy, .npz and .safetensors. But read the caveat: unsupported GGUF quantization formats are cast to float16 on load. A k-quant doesn't execute natively under MLX — it decompresses, and the memory saving evaporates.

The benchmark problem

The most-cited head-to-head numbers come from Ollama, which moved its Apple silicon path onto MLX in preview on 2026-03-30. Their published figures, tested on M5 hardware: prefill 1,154 tok/s on 0.18 versus 1,810 on 0.19; decode 58 versus 112 tok/s. That looks like a decisive format win. It isn't. Version 0.18 ran a Q4_K_M GGUF and 0.19 ran NVFP4 through MLX — engine, version and quantization scheme all changed at once.

The preview also carried hard limits: a Mac with more than 32 GB of unified memory, and only one accelerated model at first. Ollama has kept investing — a 2026-06-11 update added NVIDIA's NVFP4 4-bit format, up to 20% faster output through fused Metal kernels, and a snapshot system for agent workloads, with the claim that NVFP4 roughly halves the quality loss of 4-bit quantization relative to unquantized BF16. Still not a clean A/B. I couldn't find any first-party benchmark isolating GGUF against MLX at matched bit-width on identical hardware, so treat every "MLX is X% faster" number as confounded.

Who should pick which

Pick GGUF if you run models on more than one kind of machine, if you want whatever came out this week, if you have an Intel Mac or a pre-Sonoma macOS install, or if you want the extreme low-bit end of the curve with imatrix calibration. It's also the safer cross-platform default: llama.cpp has bindings for nearly every language you'd write in.

Pick MLX if you're on Apple silicon and staying there, if you want to fine-tune rather than only infer, if you're building a native app in Swift, or if you want the learned-quantization methods clawing back quality at 2–4 bit.

Pick neither, directly, if you just want to use a model. Neither GGUF nor MLX is a product a normal person can use — no installer, no GUI, no model picker, no update mechanism. Reaching a working chat window means assembling a runtime like Ollama or LM Studio, or writing code. Outlier is one example of the other approach: an Apple-silicon-only Mac app with seven fixed tiers where the format question is decided for you and never surfaced. You pick Nano or Core, not a quant type. That's a genuine trade — the 192,308-model catalog and every knob above, given up in exchange for not touching any of it. If turning those knobs is the fun part, run the frameworks directly. It costs nothing.