Turning off ‘thinking’ made our local model better at math: 46/60 vs 33/60
- On our smallest shipping model, turning thinking off made it better at maths: 46/60 correct against 33/60, or 76.7% against 55.0%.
- That is +21.7 points at z = 3.52 on GSM8K, and it finished 36% faster at the same time.
- The cause is not philosophical. The model had a 400-token budget and its “let me consider…” preamble was spending 200–300 of them before it reached the arithmetic.
- So the rule is narrow: when the reasoning competes with the answer for the same budget, the reasoning loses. On a model with room to think, expect the opposite.
The received wisdom is that making a model think out loud helps it with maths. Chain-of-thought is one of the most replicated results in the field, and every reasoning model ships on the strength of it. So when we turned thinking off on our smallest tier and watched the maths score go up by more than twenty points, the first assumption was that the benchmark was broken. It was not. The explanation turned out to be mundane, mechanical, and much more useful than the headline.
What we ran
Two configurations, identical in every other respect: the same GSM8K problems, the same production weights of our Nano tier, greedy decoding both times so there was no sampling luck to argue about. The only difference was the chat template’s native enable_thinking switch. We ran it first at n = 30, saw a gap large enough to distrust, and re-ran it at n = 60 to find out whether it survived.
| Setting | Correct (GSM8K, n=60) | Accuracy | Wall time | Verdict |
|---|---|---|---|---|
| Thinking on, greedy | 33 / 60 | 55.0% | baseline | — |
| Thinking off, greedy | 46 / 60 | 76.7% | −36% | ← Winner |
It survived. +21.7 points, z = 3.52 — about three and a half standard deviations, which is not a result you get from thirty coin flips going your way. And it was faster, which is the part that should make you suspicious in a good way: the cheaper configuration won. Usually you pay for accuracy.
The n=30 run, including the part that failed
The earlier pass is worth showing because it tested something else at the same time and that something else did nothing. Self-consistency — sampling the model five times and taking the majority answer — is the standard way to buy accuracy with compute. Here it is against the free change:
| Configuration | Correct (n=30) | Accuracy | Generations spent |
|---|---|---|---|
| Thinking on, greedy | 19 / 30 | 63.3% | 30 |
| Thinking off, greedy | 24 / 30 | 80.0% | 30 |
| Thinking on, self-consistency ×5 | 22 / 30 | 73.3% | 150 |
| Thinking off, self-consistency ×5 | 23 / 30 | 76.7% | 150 |
Read the bottom two rows against the second. Five times the compute bought nothing — 23/30 against 24/30 for plain greedy with thinking off. Once the preamble stopped eating the budget, the extra samples had no error left to average away. That is worth knowing before you spend five times the electricity on a local model.
Why it happens, which is the whole point
Here is the mechanism, and it is boring in the way real explanations usually are. Our Nano tier answers inside a 400-token budget. With thinking enabled, the model opens with two to three hundred tokens of “let me consider what the question is asking…” before it touches the sum. That preamble is generated into the same budget as the answer. So the model reasons its way carefully up to the edge of the cliff and then runs out of room at precisely the moment it should have produced a number.
Turning thinking off did not make the model cleverer. It stopped it spending its allowance before arriving at the shop. That is why the win is not transferable as a slogan: on a model with a generous budget, or on a task where the chain of reasoning genuinely does the work, the same switch should cost you accuracy rather than buy it. The finding is about a collision between two things sharing one budget, not about whether reasoning is good.
It also generalised further than maths on our tier: factual accuracy went from 91.7% to 100% under the same change, for the same reason.
fast_batch_15 at n=30 (63.3% → 80.0%, +16.7pp, 1.2s faster per query); confirmation fast_batch_18 at n=60 on 2026-05-28 (55.0% → 76.7%, +21.7pp, z=3.52, −36% wall time). Switch is the Qwen3-Next chat template’s native enable_thinking=False kwarg. Figures are ours and measured, not quoted from a vendor.How to decide for your own setup
- Is your output budget tight? If the model has a few hundred tokens to answer in, a visible reasoning preamble is competing with your answer. That is the case where turning it off can win outright.
- Is the reasoning the product, or the overhead? For a multi-step proof you want the chain. For “what is 17% of 340” you want the number, and the chain is rent.
- Can you run thirty of your own prompts both ways? That is an afternoon, and it beats every blog post including this one. Our result is ours; the mechanism is what should travel.
Frequently asked questions
Does turning off thinking mode always make a model better at math?
No, and our own result says why not. The gain came from a specific collision: the model had a 400-token answer budget, and its thinking preamble spent 200 to 300 of those tokens before it started the arithmetic. Remove the preamble and the budget reaches the answer. On a model with room to reason, or a task where the reasoning is the point rather than overhead, you should expect the opposite. Measure it on your own prompts before you trust either direction.
Why would less reasoning produce a better answer?
Because the reasoning is not free. On a small local model the visible chain of thought is generated into the same output budget as the answer, so a long preamble competes with the result rather than supporting it. In our run the model would talk itself through the setup and then run out of room at the point where it should have produced the number. Turning thinking off did not make the model smarter; it stopped the model spending its budget before it answered.
Does this mean larger reasoning models are pointless?
No. This is a measurement about a small model under a tight token budget, not a claim about reasoning in general. Frontier reasoning models are given far more room to think and are trained for it. What our result does show is that a technique borrowed from large models can invert on a small one, which is why every tier setting in Outlier is chosen by running the benchmark rather than by copying what the big models do. See what AI reasoning actually is.
Try Outlier free
Free Nano + Lite — local, private, no account. Pro is a one-time $249 and adds everything (all 6 model tiers incl. Plus 397B). Founders Lifetime is $249 once. Apple Silicon only.
Download for Mac