What is a paged MoE inference engine

A paged MoE inference engine runs Mixture-of-Experts models without holding every weight in memory at once. It leans on one fact about MoE: only a handful of experts fire per token. So it parks the active ones in RAM and pulls the rest off disk when they're needed. That's the whole reason a 397B-parameter model fits on a 64 GB Mac.

What MoE actually is

Start with a normal model. A standard ("dense") network runs every token through every parameter. A 70B dense model burns 70B parameters' worth of compute on each token, every time, no exceptions.

MoE breaks that pattern. A Mixture-of-Experts model chops each layer into a bunch of small "expert" sub-networks and adds a router. The router looks at the token, picks k experts (usually 2 or 8) from the pool, and only those k do anything. Everyone else sits idle for that token.

Take Qwen3.5-397B-A17B. 397B total parameters, but only 17B active per token. That "A17B" is your compute bill per token, roughly what a 17B dense model costs. The "397B" is how much the thing actually knows.

The opportunity for paging

So if only 17B of those 397B parameters do anything on a given token, why hold all 397B in RAM? You don't have to. What you actually need is:

• The shared backbone (attention layers, embedding, the router itself).
• A cache of recently-used experts, sized to whatever RAM you've got.
• An on-disk store holding all the experts, ready for the cache to pull from.

Operating systems have done this forever. It's virtual memory: don't keep the whole working set resident, page things in when you need them. The pages here just happen to be entire expert tensor blocks, usually a few hundred MB apiece.

The engine, step by step

Walk through one generated token:

1. Embed the new token, run the attention layers (always resident).
2. Router picks k experts to use for this token's MLP.
3. For each chosen expert, check the in-RAM cache. Hit → use it. Miss → read it from SSD.
4. Compute the expert MLPs, combine, produce next-token logits.
5. Sample the next token, repeat.

Where does the time go? Cache hits are basically free, just a memory access. Misses are bound by SSD bandwidth. Misses cost far more than hits because they wait on the SSD. Run enough tokens and the hot experts settle into cache, so the average read cost keeps dropping.

The numbers, in practice

Numbers from Outlier's V9 paged engine running Qwen3.5-397B-A17B on a 64 GB Mac Studio M1 Ultra:

• Peak OS-level RSS: about 14 GB (the capacity engine would want ~209 GB plus headroom for the same model)
• Decode speed: about 1.59 tokens per second
• SSD read pattern: concurrent pread with queue depth 24

1.59 tok/s is a measured number, not a brochure figure. It comes from the K=20 sweep of 2026-05-01 (5 prompts of 80 tokens, fresh process) on an M1 Ultra with 64 GB. Sure, it's slower than the in-RAM capacity engine. But that engine wants 256 GB of RAM, which most people simply don't have. The alternative isn't a faster run. It's no run.

Where paged MoE wins and loses

Wins:

• Runs models that flat-out wouldn't fit on your hardware otherwise.
• The memory footprint stays bounded, so the rest of your machine is still yours to use.
• Cache locality means your common usage patterns stay quick.

Loses:

• Cold expert misses hit the SSD, not memory. Your first response is the slowest one.
• Dense (non-MoE) models get no benefit here. They genuinely touch every weight on every token.
• An external SSD over Thunderbolt runs about half the throughput of internal NVMe, so paging slows down to match.

Why this matters for local AI

This changes who gets to run these models. Before paged MoE, a flagship-class MoE model meant a Mac with enough unified memory to hold all of it, far more than 64 GB. Now the same model runs on a 64 GB Mac Studio. That architectural trick is the reason Plus 397B is something an individual developer can actually run, not just a well-funded lab.

Questions about a paged MoE inference engine