Unified memory vs dedicated VRAM for LLM inference

Disclosure first: I build Outlier, a local AI app that only runs on Apple Silicon, so I have an obvious interest in one side of this. I've stuck to published specs and public benchmark tables, and labeled every number I couldn't verify at the source. Where a discrete GPU wins, it wins.

One physical fact everything follows from

NVIDIA's own inference-optimization guide is blunt: for token generation, the time spent moving weights, keys, values and activations out of memory dominates the time spent computing. NVIDIA calls it "a memory-bound operation."

That splits local inference into two problems. Prompt processing — reading your question, your files, your codebase — is compute-bound. Token generation — the words appearing on screen — tracks memory bandwidth far more closely than TFLOPs. Any comparison quoting one number hides half the story.

On Apple silicon the CPU and GPU share one pool — MLX's docs note arrays live in shared memory, usable by any device without transferring data. A discrete GPU has its own isolated, much faster pool, and everything must cross PCIe into it.

Capacity: what actually fits

Here unified memory sits in a different class. Apple's M3 Ultra configures to 512GB at over 800GB/s (819GB/s per the Mac Studio specs), and Apple claims models over 600 billion parameters run on device. DGX Spark ships 128GB of coherent memory at 273GB/s for inference up to 200B. An RTX 5090 has 32GB; the RTX PRO 6000 has 96GB with ECC.

Those are hard ceilings. Spill out of VRAM and the offloaded layers cross PCIe at a fraction of GDDR7 bandwidth — throughput falls off a cliff. Unified memory degrades gently instead. Here's what my app's tiers need:

Everything through 35B fits a 5090. The 397B tier doesn't, at any quantization worth running; on a 64GB Mac it works only because a paged mixture-of-experts loader streams experts from SSD, holding peak resident memory near 11GB at 2.1 tokens per second on an M1 Ultra. Slow, but it runs. llama.cpp's DGX Spark file shows the same shape on a mainstream model: 59 GiB of gpt-oss-120b at roughly 2,444 tokens/sec prefill, 59 tokens/sec generation. On a 5090 it doesn't load.

One caveat: unified memory is shared. The model competes with macOS, your browser and the app, and Metal caps GPU allocation through a documented working-set limit — the widely repeated 75%-of-RAM figure is community-reported, not published by Apple. Usable budget runs well under the sticker number.

Bandwidth: speed once it fits

llama.cpp keeps two community benchmark threads in its repo, same model (LLaMA 7B Q4_0). Apple silicon: M4 Max 714 tokens/sec prefill and 70 tokens/sec generation; M2 Ultra 1,238 and 94. CUDA: RTX 5090 14,073 and 290; RTX PRO 6000 14,855 and 274.

The arithmetic is mine, not theirs: a 5090 is about 20x faster than an M4 Max at prompt processing but only about 4x faster at generating tokens. That's the split above — compute-bound prefill, where discrete GPUs are an order of magnitude ahead, and bandwidth-bound decode, where the gap collapses toward the bandwidth ratio.

Apple is closing the compute half deliberately: M5 puts a Neural Accelerator in every GPU core and claims over 4x M4's peak AI compute, with M5 Pro and M5 Max at 307GB/s and 614GB/s. Right fix — but it's a 20x gap, not a 2x one.

Side by side

Thermals, power, and sustained load

An RTX 5090 draws 575W for the card alone, with an 850W supply recommended; the PRO 6000 is rated 600W. An entire M3 Ultra Mac Studio — CPU, GPU, memory, storage, fans — is rated 480W maximum continuous. DGX Spark's GB10 is 140W. That shows up as noise, heat, and where the machine can live — and for a laptop there's no dedicated-VRAM option in this class at all.

Price per usable gigabyte

Where official prices exist: DGX Spark's Founders Edition went from $3,999 to $4,699 in February 2026, which NVIDIA attributed directly to industry-wide memory supply constraints — about $37 per gigabyte of pool. The 5090's $1,999 launch MSRP is secondary-sourced, roughly $62/GB. Secondary reporting puts the 96GB PRO 6000 at $8,565 at launch, about $89/GB, and higher on NVIDIA's marketplace by mid-2026.

I won't quote a Mac Studio figure — Apple's store pages don't render prices for verification, and I'd be guessing. Tom's Hardware reports Apple withdrew the 512GB option and raised the 96GB-to-256GB upgrade to $2,000 during the shortage. Date every price you read, mine included — TrendForce forecasts server DRAM contract prices up another 13–18% quarter over quarter in Q3 2026.

Where dedicated VRAM genuinely wins

If your work looks like this list, buy the GPU.

• Time to first token. About 20x on identical llama.cpp runs. For agentic coding or long-context RAG — anything re-reading a big codebase each turn — that's the difference between instant and painful.
• Batching and concurrency. The compute headroom idle during single-stream decode is what serves many requests at once. A Mac serving one user leaves little on the table; a Mac serving eight falls apart.
• Ecosystem depth. vLLM lists CUDA, ROCm and Intel XPU as GPU platforms; Apple silicon GPU support lives in a separate vllm-metal repo. TensorRT-LLM and new kernels land on CUDA first and Metal later, if ever.
• Training and fine-tuning. NVIDIA markets the PRO 6000 for local fine-tuning and DGX Spark up to 70B. The Mac story is overwhelmingly an inference story.
• ECC and repairability. The PRO 6000's 96GB is ECC-protected; Apple's isn't. You can add a card, swap one, or rent the same GPU in a cloud. Apple silicon memory is soldered — what you buy is what you have forever.

And NVIDIA has conceded the capacity argument by shipping its own unified-memory box — the question is drifting from Mac versus GPU toward CUDA versus Metal on similar memory architectures.

Who should pick which

Pick unified memory if you want to run large models at all on one machine, you value quiet and low power, you work on a laptop, or your workload is single-user chat, writing and coding.

Pick dedicated VRAM if your models fit in 32–96GB, time-to-first-token is what you feel most, you're serving multiple users, or you plan to fine-tune.

Both paths run fully on-device, so cloud privacy questions don't apply to either; the weights' own licenses are the only policy layer. If you land on Apple silicon, prefill is your weak axis — prompt caching and prefix reuse buy more than any spec bump. That's the bet Outlier makes: Apple Silicon only, macOS 12+, no Windows or Linux build, no live web search.