What is AI reasoning? Thinking models explained

"Reasoning" is the word the industry landed on for a fairly specific behavior: the model writes out its working before it answers. Sometimes you see that working, sometimes it's hidden behind a spinner. Either way it's the same trick, and it's a genuinely useful one — as long as you know what it costs and when it doesn't pay.

What a reasoning model actually does

A reasoning model — you'll also see "thinking model," they mean the same thing — produces a chain of thought before the final answer. Ask it a word problem and instead of emitting an answer immediately, it emits a stretch of text where it restates the problem, tries an approach, notices a mistake, backs up, and only then writes the answer you asked for.

The examples you've probably met: OpenAI's o-series; DeepSeek-R1, a 671-billion-parameter mixture-of-experts model released with open weights; and the Qwen models, which ship a thinking mode you can switch on and off. That last one matters, because a toggle turns "is thinking better?" from an opinion into something you can measure.

Under the hood, it's still next-token prediction

Here's the part the marketing tends to skip. There is no separate reasoning module bolted onto these models. It's the same architecture doing the same thing it always did — predicting the next token, one at a time — described in the large language model explainer.

What changed is training and prompting. The model has been taught to spend tokens working the problem before it commits to an answer. And because every token it generates is another forward pass through the network, a long stretch of working means substantially more computation spent on that one question.

That's the whole mechanism: more tokens equals more compute per question. It isn't a new kind of cognition. Which is also why it isn't free, and why it isn't automatically better.

Thinking tokens are real tokens

Every token in that chain of thought has to be generated the same way any other token is. The consequences are concrete:

• In the cloud, you pay for them. Money and latency, whether or not the provider shows you the chain. A hidden reasoning trace is still a billed one.
• On your own machine, you pay in seconds and battery. There's no invoice, but there is a fan, a warm laptop, and a wait before the first useful word appears.

None of that is a reason to avoid thinking modes. It's a reason to spend them where they buy something.

I measured it on my own model, and thinking lost

I run a local model on Apple Silicon, so this is easy for me to test rather than argue about. I took Outlier's Nano tier, ran 30 GSM8K grade-school math problems through it with greedy decoding, and flipped the thinking mode on and off. GSM8K is exactly the kind of multi-step arithmetic that chain of thought is supposed to be built for.

Thinking mode was worse by 16.7 percentage points, and thinking OFF was 32% faster. I'll be careful about what that does and doesn't prove. It's one model, one benchmark, one decoding setting. It is not evidence that reasoning modes are a scam, and I'm not going to pretend I know exactly why this particular model degrades when it talks to itself.

What it is: a measured counterexample to "thinking is always better." That claim gets repeated as though it were settled, and on my hardware, on a task purpose-built for chain of thought, it was false. If you're choosing a mode for real work, measure your model on your task. Nobody's benchmark is a substitute for that.

Where thinking earns its tokens

The pattern I keep seeing is that chain of thought pays when the problem has dependent steps — where getting step four right requires having written down step three. It wastes tokens when the answer was already in the model and just needed to come out.

"Usually" is doing real work in that table. My Nano measurement was a multi-step math task in the first row, and thinking still lost. The category tells you where to look; it doesn't tell you the answer.

The practical rule: match the mode to the task

Leaving thinking on by default is the mistake I see most. It feels safe — more deliberation, surely — but it means you pay latency and battery on every message, including the dozens of trivial ones, and on at least one real pairing it also cost accuracy.

So: switch it on when you hand the model something with several dependent steps, and leave it off for everything else. If your tool exposes a toggle, that toggle is worth learning. And if you can run the same 20 or 30 prompts both ways on the work you actually do, do that once. It takes an afternoon and it settles the question for your setup permanently. A related idea — having the model verify its own output instead of just deliberating longer — is covered in how an AI agent checks its own work.

Running locally makes this easier, mostly because the feedback is immediate and nothing is metered. You can run a benchmark thirty times over without watching a bill climb. That's the argument in what local AI is, and testing your own model is a good example of why it matters.

Frequently asked questions