What is a token in AI?

Tokens are one of those pieces of jargon you can ignore right up until the moment you can’t. Every limit you’ll ever hit is denominated in them: what an API charges, how much of your conversation the model can still see, where a rate limit cuts you off. So it’s worth ten minutes to be able to convert “my 40-page report” into a number you can actually check against a limit.

Why not just count words?

Because a model has to handle text it has never seen — a surname, a typo, a variable called usr_cfg_2 — and a fixed word list can’t. Tokenisation splits text into pieces from a fixed vocabulary, typically tens of thousands of entries, chosen so that common words survive as single pieces and rare ones break into fragments.

So the model doesn’t see letters, and it doesn’t quite see words. It sees a sequence of IDs from that vocabulary. When it writes, it emits one of those IDs at a time and the app stitches them back into text on your screen.

This also quietly explains a classic party trick failure. Ask a model how many r’s are in “strawberry” and it may get it wrong — not because it can’t count, but because it never saw the individual letters. It saw two or three chunks.

Roughly how many tokens is your text?

Rules of thumb, for English:

Treat these as estimates for planning, not for billing. If a precise number matters, run the text through the tokeniser for the specific model — different model families split text differently, so the same paragraph genuinely has different token counts depending on who made the model.

The three limits tokens actually control

Context window. The maximum tokens the model can hold at once — your system instructions, the whole conversation so far, and the answer being written. When a conversation outgrows it, the oldest parts get dropped or summarised, which is why AI forgets things you said earlier.

Price, if you’re using an API. Billed per token, usually with input and output priced differently. This is why a long document pasted into every message is expensive: you pay for all of it, every turn.

Output length. A separate cap on the reply itself. It’s the reason answers sometimes stop mid-sentence — the model didn’t lose its train of thought, it ran out of allowance.

Tokens are also the speed unit

Generation speed is quoted as tokens per second, which is why the number feels abstract until you convert it. A comfortable adult reading pace is somewhere around 4–7 tokens per second, so anything above that arrives faster than you can read.

On my own M1 Ultra, our smallest tier runs at 71.7 tok/s and the 27B at 20.7 tok/s — both well past reading speed. The 397B tier manages 2.1 tok/s, which is genuinely slow enough to notice and is a deliberate trade for running a model that large on a desktop at all.

Does any of this change when the model runs locally?

Tokenisation, context windows and output caps work identically. What changes is the economics: there’s no per-token bill, so a long document costs you time and memory rather than money.

Memory is the real constraint instead. Every token in context occupies space in a cache that grows as the conversation does, which is why a very long chat on a small machine eventually slows down or runs out of room. Not a meter — physics.

Practical habits worth having

• Estimate before pasting. Words × 1.33 is close enough to know whether a document fits.
• Don’t paste what you don’t need. Irrelevant text pushes the useful part further from the answer and burns the window.
• Start fresh for a new task. A new conversation resets the token count; continuing an old one carries all of it forward.
• Expect code to cost more. If you’re near a limit with source files, you’re nearer than the word count suggests.