What is a system prompt, and why does it matter?

A system prompt is the block of instructions that goes at the very top of a conversation, before you've said anything. It tells the model who it's being, what rules apply, what format to answer in, and what not to do — and it applies to the entire session, not just the next reply. Most people meet it as a text box labeled "custom instructions" and never think about it again. It's worth thinking about, because it's doing more work than any single question you'll ever type.

What a system prompt actually is

Strip away the interface and a conversation with a language model is one long string of tokens. Your messages are in there. The model's replies are in there. And at the very front sits the system prompt: persona, rules, output format, constraints. Everything after it is read in its light.

That's the whole mechanism. No hidden config file, no separate API the model consults. It's text, in the same window, in front of everything else. Slightly deflating to learn, and also the most useful thing to know about it — once you get that it's just tokens at the top, both its power and its limits stop being mysterious.

Why the model listens to it at all

If it's just more text, why does the model take it more seriously than a passing remark in message four? Two reasons, and neither is magic.

The first is position. It's at the top. Everything the model reads afterward is conditioned on what came before, so instructions in front of the conversation shape the whole conversation by construction. The second is training. Models are trained on conversations where the leading block is treated as standing instructions, so they've learned to weight it that way.

The honest consequence: a system prompt is strong steering, not a hard switch. There's no privileged channel enforcing it. If you write "never use bullet points" at the top and then paste something extremely bullet-shaped, you may still get bullets. Treat it as the strongest lever you have, not a guarantee, and you'll set your expectations correctly.

The other reason it matters: speed

Here's the part almost nobody mentions, and it's the thing I spend my days on. Because the system prompt sits at the top and rarely changes, it's the ideal thing for a runtime to cache.

Every request, a model has to process the tokens in front of it before it can produce the first word of a reply. If the first several thousand are identical to last time, redoing that work is waste. So don't: prefill the system prompt once, keep the result, reuse it on every request after.

In Outlier, doing exactly that — prefilling the system prompt once per model tier and reusing it across requests — took follow-up time-to-first-token from about 18 seconds to about 1.3 seconds. Roughly 13x. Same model, same prompt text, not one word changed. All of the gain came from not repeating work on a block that hadn't changed. The mechanics of that reuse are covered in what prefix caching is.

The corollary that bit us

If a stable top-of-conversation is worth 13x, then anything that mutates it between turns throws that away. I learned this the direct way.

Outlier has a memory feature that recalls relevant facts and injects them into the conversation. It was running its recall against the most recent user message — which sounds right, and is completely wrong for caching. Every step of an agent loop produced a new "most recent message," so the recalled facts changed, so the injected block changed, so the system block changed. Prefix reuse fell to zero. Every single step re-prefilled the whole conversation from scratch.

The fix was one decision: recall against the first user message instead. The block became stable, and the reuse held. Nothing about the model changed. The lesson generalizes past our codebase — if your top-of-conversation block is dynamic, you're paying full price on every turn, and you probably can't see it. It looks like the model being slow. It's the plumbing.

System prompt or user message? A simple split

Practical version for anyone actually using one of these apps: durable instructions go up top, per-question detail goes in the message. And once the top block is set, leave it alone.

A rewritten system prompt every session isn't a tuned setup, it's a moving target — for the model and for the cache underneath it. Write it once, write it carefully, then let it sit. For the writing part, how to write a good AI prompt covers the rest.

Who owns that layer

Cloud AI apps ship their own hidden system prompts. You don't see them, you can't edit them, and they can change between the Tuesday you built a workflow on one and the Wednesday it stops behaving the same. Your custom instructions get appended to something you were never shown.

Running locally, that layer is yours. You write the block, you know what's in it, and it doesn't change unless you change it. I won't claim that alone makes local worth it — the honest trade-offs live elsewhere. But if an app's behavior has ever shifted underneath you with no changelog, owning your own instructions is a real thing to own.

Frequently asked questions