Apple MLX vs llama.cpp: picking a local inference runtime on Mac

Disclosure up front: I build Outlier, a local AI app for Mac, and it runs on MLX — so I'm not neutral. What follows is the comparison I ran before choosing, including where llama.cpp is the better engineering call and my own app would have been easier to ship.

What each one actually is

MLX is an array framework for machine learning on Apple silicon, published by Apple's machine learning research group under an MIT license. It's NumPy-shaped: lazy evaluation, composable function transforms, autodiff, and Python, C++, C and Swift APIs. mlx-lm is the language-model layer on top — generate/chat/convert CLIs, a streaming Python API, LoRA and full fine-tuning including on quantized models, and distributed inference through mx.distributed.

llama.cpp is LLM inference in C/C++, also MIT, with no dependencies at all and a stated goal of minimal setup across a wide range of hardware. Apple silicon is a first-class target — ARM NEON, Accelerate and Metal — but so are CUDA, HIP, Vulkan, SYCL, MUSA, OpenCL, WebGPU and x86 with AVX/AVX2/AVX512. On macOS, Metal is on by default; you turn it off with -DGGML_METAL=OFF at build time or --n-gpu-layers 0 at runtime.

Neither ships model weights, neither has a pricing page, and both are fully local — the only network traffic in the default path is pulling weights from Hugging Face. Neither documents telemetry or publishes a privacy policy, so that's absence of evidence, not a promise.

The mechanism: unified memory, and where the time goes

MLX's central design decision is that arrays live in unified memory shared by CPU and GPU. There's no host-to-device copy step, and device placement is per operation rather than a whole-model commitment. On a Mac, where both already share one pool, that matches the hardware instead of emulating a discrete GPU that isn't there.

Inference also has two phases with different bottlenecks. Apple's research states it plainly: the first token is compute-bound; subsequent tokens are bounded by memory bandwidth. Measuring MLX on an M5 MacBook Pro against a comparable M4 in November 2025, Apple saw time-to-first-token improve 3.33x–4.06x but token generation only 1.19x–1.27x, tracking the bandwidth jump from 120 GB/s to 153 GB/s. That's a hardware comparison, not a runtime one, and it gets misquoted as one.

GGUF and MLX are not interchangeable

llama.cpp uses GGUF, with integer quantization from 1.5-bit to 8-bit and conversion tooling in-tree. MLX uses its own format via mlx_lm.convert, defaulting to 4-bit, with nvfp4 added in v0.32.0. A GGUF won't load in MLX, and an MLX repo won't load in llama.cpp.

That decides your catalog. llama.cpp's README lists 50-plus text model families plus multimodal; mlx-lm points at thousands of Hub models with pre-quantized conversions in the mlx-community org. In practice a brand-new architecture gets a GGUF first, and llama.cpp's release pace means support often lands within days.

Serving and embedding: the real gap

mlx-lm ships an OpenAI-shaped HTTP server with /v1/chat/completions and /v1/models, and its own docs carry a warning worth quoting: it is not recommended for production as it only implements basic security checks. It's a development convenience, not a serving layer.

llama-server is a different class of thing. Per its README: OpenAI-compatible chat, responses and embeddings routes, Anthropic Messages-compatible completions, reranking, parallel decoding with multi-user slots, continuous batching, multimodal, schema-constrained JSON, tool calling for effectively any model, speculative decoding, LoRA adapters, prompt caching with KV reuse, API-key auth, and a web UI. If you need a server this quarter, that list is the whole argument.

Embedding splits the same way. llama.cpp compiles to small self-contained binaries and exposes libllama, with 30-plus language bindings. MLX's Python path needs native arm64 Python 3.10+ — Rosetta breaks it — so you're bundling a Python runtime inside a notarized .app unless you move to mlx-swift. That's first-party and covers macOS, iOS and visionOS, but command-line SwiftPM can't build its Metal shaders, so the final build goes through Xcode.

Side by side

Where each one genuinely wins

MLX wins on Apple silicon throughput, and on being more than an inference engine. A reproducible third-party suite (apple-silicon-llm-bench) reports MLX-Swift ahead on decode for small and mid-size models on an M4 Max: Qwen 2.5 0.5B at 531 vs 297 tok/s, Qwen 3.5 2B at 292 vs 150, Gemma 4 E2B at 185 vs 119. Ollama's March 2026 numbers after moving its Apple path to MLX show decode 112 vs 58 tok/s on Qwen3.5-35B-A3B in NVFP4, with its own caveats: preview build, more than 32 GB of unified memory, few models supported. Secondary write-ups say the advantage shrinks toward parity at 27B and up, where both hit the same bandwidth wall — directional, not measured. MLX also trains; llama.cpp doesn't try to.

llama.cpp wins on reach, completeness and shipping friction. MLX can't produce a Windows build, can't drive a discrete GPU, and can't run on an Intel Mac; llama.cpp covers all of that from one codebase. Its server is close to finished software rather than a demo, GGUF is the de-facto distribution format for open weights, and zero dependencies makes notarizing a Mac app far less painful than shipping a Python stack. Five thousand-plus releases mean bugs on new architectures get fixed in days — though that pace also brings API churn and no LTS branch.

They aren't mutually exclusive. LM Studio's docs describe shipping llama.cpp variants alongside an Apple-silicon-only MLX engine, and Ollama went MLX-first then, per its June 2026 post, brought GGUF back through llama.cpp for breadth.

Who should pick which

Pick llama.cpp if you ship on more than one platform, need a server with auth and batching now, want the widest model catalog, have to support Intel Macs, or want the smallest binary inside a signed app.

Pick MLX if you're Apple-silicon-only by design, want the measured decode throughput on small and mid-size models, need fine-tuning in the same framework as inference, or are building a native Swift app for Mac and iPhone from one code path.

Pick both if different models in your product want different engines and you can afford two paths.

For Outlier the constraints decided it. It's Apple Silicon only, M1 or later — no Intel, no Windows, no Linux, no mobile app — so llama.cpp's portability, its strongest card, buys me nothing. MLX gave me room to build what isn't in either runtime: the Plus tier is a 397B-a17b MoE streaming experts from SSD at roughly 11 GB peak RSS, which needed paging control no general-purpose engine hands you. The trade is real — I wrote my own serving layer, prompt caching and model manager, all of which llama-server would have given me free.

One last thing, and it's the most useful advice here: the author of apple-silicon-llm-bench warns about session-to-session variance and recommends benchmarking the exact artifact you ship, since quantization method and build flags move the numbers as much as runtime choice does.