Measured on our development Mac (Apple Silicon, on-device coding tier). Same answers — only the waiting changed.
A local model waits before answering because it has to read its prompt first — and most of that prompt is a big, fixed instruction sheet that never changes. A naive setup re-reads it from scratch on every question. We taught Outlier to prepare that fixed part once and reuse it, so after the model is warm, replies start almost immediately.
When you send a message, the model doesn’t start writing right away. First it reads everything it’s been given — your message, the conversation so far, and a hidden system prompt: a couple thousand words of standing instructions about how to behave, format code, use tools, and so on. Reading all of that is a step called prefill, and on a compressed (quantized) model running on your own hardware, prefill of a large system prompt can take many seconds. The actual writing that follows is fast; the wait you feel is almost entirely prefill.
The maddening part: that system prompt is identical every single time. Re-reading it for every new question is like re-reading the rulebook of a board game before every turn.
You should — and that’s the fix. But on modern efficient local models it’s trickier than it looks. Many of today’s fast on-device models are hybrid: they mix ordinary attention layers with linear-attention or state-space layers that keep a small, compressed running summary instead of remembering every token individually. That design is a big reason they’re fast and fit in memory — but it has a catch. You can’t “rewind” that running summary to an earlier point. So the tempting trick — take a full conversation’s memory and trim it back to just the system-prompt part — simply doesn’t work; there’s nothing to trim back to.
The approach that does work is to prepare the stable system-prompt state on its own, once, and then reuse that prepared state for every new question — only reading the small, changing part (your actual message) fresh each time.
There was one more twist. Our first attempt still re-read everything, and it took careful measurement to see why: the “fixed” system prompt wasn’t entirely fixed. It quietly included a few personal memory notes, chosen based on what you asked — so the supposedly-stable part actually changed a little with every question, which broke the shortcut. The fix was to remember only the genuinely-unchanging instructions and treat the personal notes as part of the small, fresh piece. After that, the shortcut hit every time.
Two things worth being straight about. First, the very first question in a session still has a one-time cost to load the model into memory — the speed-up is for every question after it’s warm. Second, the numbers above are from our own machine; your exact timing depends on your Mac and which model tier you run. The shape of the win — stop re-reading the fixed instructions — holds regardless.
Responsiveness is most of what makes an assistant feel good to use. A local model that’s private and free but makes you wait 18 seconds per question loses to a cloud tool that answers instantly. Closing that gap — on-device, with nothing leaving your Mac — is what makes a local assistant one you actually keep using.
Download Outlier for MacRequires Apple Silicon (M1–M4). macOS 12+. Runs on your Mac; nothing leaves the device unless you enable cloud escalation.
Outlier runs on your Mac. macOS 12+ on Apple Silicon. Back to home.