Can the Apple Neural Engine run an LLM? I measured it

Every Mac spec sheet advertises a Neural Engine. Mine has sixteen cores. So when I started building local AI on Apple silicon, the obvious question was why that hardware sits essentially unused while the GPU does all the work generating text — and whether I could put it to work. I spent a session finding out, and the answer surprised me in a specific way: the thing everyone assumes is the problem turned out to be fine, and the real blocker is somewhere nobody points at.

What I was actually trying to build

The idea was speculative decoding. You run a small, fast "draft" model to guess several tokens ahead, then have the big model verify the guesses in one pass. When it works you get a real speedup. On a Mac it tends not to work as well as advertised, because the draft and the verify both want the GPU and end up fighting each other for it.

Hence the appeal of the Neural Engine. If the draft model could live on the ANE — idle silicon, sitting right there — the GPU would be left alone to verify, and the draft would be close to free. That was the hypothesis. It is a good hypothesis. It is also wrong, and it is worth explaining exactly how.

How to tell where an operation actually runs

This is the part I'd want if I were reading someone else's writeup, so I'll put it before the results.

Most claims about ANE usage are inferred from power draw, which is a proxy and a weak one. There's something better. Core ML exposes MLComputePlan, and specifically get_compute_device_usage_for_mlprogram_operation, which reports the device assignment for every individual operation in a compiled model. Not an estimate. Apple's own planner telling you where each op landed.

That distinction matters enormously here, because the failure mode I found is silent. Nothing errors. Nothing warns. Your model loads, runs, produces correct output, and reports success — while executing entirely on the wrong processor. Without a per-op device map you would simply conclude that the ANE is slow, and you'd be wrong: it isn't running at all.

The cliff at 16 layers

I built a representative decode step — attention output projection plus a SwiGLU MLP, hidden size 2560, intermediate 6912, fp16 — and stacked it to increasing depths, checking the device map at each one.

Up to 8 layers the scaling is clean and linear, about 2.9 ms per layer, every operation on the Neural Engine. Then it falls off a cliff. Not a partial split, not a graceful degradation — at 16 layers the planner gives up and assigns the entire graph to the CPU.

A 32-layer model is the shallow end of what counts as a real language model. At that depth we're looking at 139 to 242 milliseconds per token, which is somewhere around 4 to 7 tokens per second, on a processor that was never the point. For comparison, the smallest tier I ship runs at 71.7 tokens per second on the same machine using the GPU.

The handoff was never the problem

Here's the part I got wrong going in, and I think most people have it wrong too.

The standard explanation for why nobody runs LLMs on the ANE is the cost of moving data — you have to marshal tensors out of your GPU framework, hand them to Core ML, and get them back, once per token. That sounds expensive. I measured it in isolation with a trivial identity model, so the number is pure overhead with no compute in it: 0.84 ms median, with a 10th percentile of 0.50 ms and a 90th of 1.70 ms.

Under a millisecond. And in any real speculative-decoding loop you'd draft a chunk of several tokens per call, so that cost amortises down to a fraction of a millisecond per token. The handoff is a rounding error. It is not why this doesn't work, and if you've been repeating that explanation — I was — it's worth dropping.

Why the ANE loses even when it wins

There's a second, more fundamental problem, and it survives even if you solve the residency one.

Take the clean linear scaling from the shallow runs and extrapolate it. At 2.9 ms per layer, a 32-layer draft that stayed on the Neural Engine would land around 90 ms per token. The GPU does the same work at roughly 0.3 ms per layer, so about 10 ms per token. Even in the fantasy where Core ML keeps the whole graph resident, the ANE is around nine times slower per token than the GPU it was supposed to relieve.

That isn't a defect. It's what the hardware is for. The Neural Engine is built for high throughput on big batched convolution-shaped work — the vision and audio models that run constantly on your phone at very low power. Generating text one token at a time is the opposite workload: tiny, sequential, latency-bound. The ANE is extremely good at a job that language model decoding simply is not.

What this means if you just want to run AI on your Mac

Practically: ignore the Neural Engine core count when you're deciding whether a Mac will run local AI well. It's not the number doing the work.

The specs that actually matter are memory capacity, which decides what size model fits at all, and memory bandwidth, which largely sets how fast tokens come out. A 16-core ANE next to 8 GB of RAM will run local models worse than a machine with fewer neural cores and 32 GB. If you're sizing a machine, our RAM guide is the more useful read, and why Apple silicon works without a discrete GPU covers what the unified memory design actually buys you.

None of this makes the Neural Engine useless. It's working hard on your Mac right now — Face ID, dictation, photo analysis, live text, background audio. It's just not the part generating your chat replies, and no amount of marketing about neural cores changes what Core ML's planner does with a deep decode graph.

What would change my answer

Two things, either of which I'd genuinely like to see:

• Core ML starts hosting deep autoregressive decode graphs on the ANE with single-token latency under about 5 ms — meaning both the residency cliff and the per-token gap close.
• A stateful-KV ANE decode path arrives where a 24-layer-plus graph stays assigned to MLNeuralEngineComputeDevice and beats the GPU per token.

Neither is true today on the hardware and software I tested. If either becomes true, the arithmetic changes and I'll re-run this.