How to speed up local AI on your Mac

Almost every "local AI is too slow" complaint I get is one of two very different problems wearing the same coat. People describe both as slow, but the causes don't overlap and neither do the fixes. Before you buy RAM or download a smaller model, work out which one you actually have.

The two numbers that matter, and why they're different

TTFT — time to first token — is how long you stare at nothing after hitting enter. tok/s — decode speed — is how fast words appear once the answer has started. A setup can be terrible at one and fine at the other. A model streaming at a comfortable 20 tokens per second still feels broken if you waited 18 seconds to see the first word.

Watch your own app for one turn and note which half hurt. Long pause, then normal text? Next section. Text crawling out once it starts? Skip to the RAM section.

Fix time to first token first

TTFT is dominated by prefill: the model has to read your whole prompt before it can write anything. And in a chat app, "your prompt" isn't the sentence you just typed — it's the entire conversation, re-fed every turn. That's why turn 20 feels slower than turn 2 even though you typed the same amount. Three levers actually move it:

• Keep the stable part of your prompt at the top and unchanged. Runtimes cache the work already done on a prefix and reuse it. Edit an early message and you invalidate that cache, so the whole thing gets re-read. Put what doesn't change — instructions, the file you're working from — first, and add new material at the bottom.
• Start a new conversation for a new task. Continuing a 60-message thread to ask an unrelated question means re-reading 60 messages of irrelevant context every turn.
• Don't paste 10,000 words you don't need. Every token in the prompt is prefill work. Paste the function, not the repo.

How much is on the table? On Outlier's own engine, fixing prefix reuse took steady-state TTFT from about 18.5 seconds to the 1.4-4.7 second range. On a 6-step agent loop, step 6 went from 66.4 seconds to 16.0 — and stopped climbing with every step. Latency became flat instead of a function of how long you'd been talking. Mechanics here: what prefix caching is.

Then fix tokens per second: make the model fit

Decode speed has one dominant cause on a Mac, and it isn't the chip. If the model exceeds free RAM, macOS swaps to SSD and decode collapses to roughly 1-3 tokens per second, with the fans coming up to tell you about it. That's not a slope. It's a cliff.

So: pick a model that fits, then keep it fitting. Closing the memory hogs is the cheapest win there is — a browser with many tabs, Docker, a VM, a video editor in the background. Any of those can be the difference between fitting and not. If you're not sure what size you can hold, how much RAM you need for local AI goes through it, and running AI on an 8 GB Mac covers the tight end.

Then pick the size on purpose: a smaller tier for interactive back-and-forth, the large one for the hard question you're willing to wait on. Here's that trade on my own machines:

Plus at 2.1 tok/s is slow, and I won't pretend otherwise — it's a 397-billion-parameter model streaming experts from disk at around 11 GB of peak memory, a thing that shouldn't run on this machine at all, running. That's a "go make coffee, come back to a real answer" tool. Nano at 32 tok/s is the chat partner. Which one you reached for explains most of your speed experience.

Symptom → cause → fix

Try turning thinking mode off

This one surprised me enough that I keep repeating it. Thinking mode spends tokens reasoning before it answers, which costs time by definition. What isn't obvious is that it doesn't always buy accuracy. On a 30-question GSM8K run against the shipping Nano model, thinking off scored 80% against 63.3% with thinking on, and finished 32% faster.

Be honest about what that is: one model, one task, n=30. Thinking isn't useless, and on harder reasoning it earns its keep. But the assumption that more reasoning tokens always mean better answers is worth testing rather than believing. Toggle it, run the same five prompts both ways, keep whichever won.

What you can't fix

A fanless MacBook Air will throttle on a long sustained generation. There's no fan to move the heat out, so the chip slows itself down. That's physics, not a bug in your setup, and no setting reverses it. Plugged in and on a hard surface instead of a couch cushion helps at the margin; shorter generations help more, because the machine never gets hot enough to care.

Likewise, a big model on a small machine is still a big model on a small machine. Paged streaming makes it run at all, which is useful, but it doesn't make it quick. Quality and responsiveness at once is a hardware conversation, not a settings one.

Frequently asked questions