Local AI and math word problems on a Mac: 12 of 20 right became 20 of 20

I typed a shopping bill into our free Lite model: four pens at $1.25, three notebooks at $3.40, "give the amount only." It said $13.85. The bill is $15.20. Lite is the 9B model in the free download. A small model asked for just the number can get the number wrong. Below is how often it did, what I changed, and what the change costs.

How I tested it

I wrote 20 short word problems: a coffee order, a discount, a trip time, a bill split three ways, ages, a unit rate. Each ends with "Give the number only" or "Give the amount only", because that is how you ask when you want a figure to paste somewhere. Two of them: a train leaves at 9:40 and arrives at 13:15, how many minutes is the trip, and an $80 jacket is 15% off, what is the sale price.

I sent each one to the installed app's chat, the normal path with no special prompt, incognito, web search off, with up to 4,096 tokens allowed per reply. A reply counted as right if the last number in it was within a cent of the true answer. I also flagged any reply where the working showed up in the answer: a thinking tag, a phrase like "the user is asking", or a reply that opened with "Okay,". Then I asked 5 ordinary questions, such as "What is the capital of Canada?", to see whether anything else changed.

The "before" run is Outlier 1.11.905. The "after" run is the installed 1.11.908, today's download. Lite is a 9B model and the RAM guide lists it as fitting a 16 GB Mac.

What it got wrong

Straight away, Lite got 12 of 20 right. These are the 8 it missed in that run.

Table 1: the 8 misses on 1.11.905

• The car answer, 6.5, is the rate printed in the question. The sum was never done.
• The age answer, 10, is how old Lee will be in five years. The question asked about now.
• The ticket answer, $39, is what 2 adults and 2 children would pay. There were 3 children.
• I cannot read a cause off the other five numbers, and I did not dig further.

What I changed

My first try was the obvious one: switch on the model's own thinking by typing /think. On a different set of 10 terse problems, on 1.11.905, Lite went from 5 of 10 right to 9 of 10. But the median reply took 62.5 seconds and wrote 3,064 tokens, and 4 of the 10 replies had the working inside them. The app's standing instructions told the model to show no working at all, and the model was fighting them.

So I changed the instruction for one case. When the app reads a message as a word problem (numbers, an operation, a question), Lite is allowed to work it out privately, and the answer itself has to stay clean. Every other message is handled as before. That is what 1.11.908 does.

Table 2: Outlier Lite on the same 20 problems

• The time is from sending the message to the end of the reply, not to the first word. For the first-word wait see time to first token.
• The 479 tokens include the working you never see. That is where the 8 seconds go.
• On 1.11.908 the 5 ordinary questions took a median of 3.4 seconds and 144 tokens, with no working in any answer. I did not time those five on the old build, so I have no before and after for them.

Why thinking first helps, and why my Nano page says the opposite

A model writes a small piece of text at a time and cannot go back. Ask for the number only and the first thing it writes is the answer. It has to produce "15.20" without having added anything up. A median reply of 3 tokens is exactly that. With room to work, it can write 4 x 1.25 = 5 and carry on. That is the usual explanation and these numbers fit it. I did not test another one.

I have a page that found the opposite on Nano: with thinking switched off, a grade-school maths set went from 33 of 60 to 46 of 60 right. The difference was the budget. There the model had 400 tokens, and its built-in preamble used 200 to 300 of them before it reached the sum. Here the working has up to 4,096 tokens and never reaches the screen. So thinking is not good or bad. It helps when it has room, and when it is kept out of the answer. See also what AI reasoning is.

What I did not ship

My first build did this for Nano and Lite together. On that build Nano got 15 of 20, took a median of 27.6 seconds, wrote a median of 810 tokens, and 2 of the 20 answers had working in them. I did not publish that build. In 1.11.908 Nano answers word problems the old way, and Core and Vision 3.8 are unchanged.

How far to trust 20 of 20

Not very far, and I would rather say so. It is 20 problems, I wrote them, and each setup ran once, so every figure above is one draw and not a rate. With 20 of 20, the 95% lower bound on the true rate is about 83%. Read it as "much better than 12 of 20", not "always right". If the number is a bill you are about to pay, use a calculator.

If you use another local app, you can run my test in ten minutes. Take 10 of your own word problems. Ask each once with "number only", and once with "work it out step by step, then give the final number". Count the right ones. I have not tried that on other apps, so I do not know which way yours will go. In Outlier, add /no_think at the end of a message to skip the thinking step. Put it at the end, not the start: a message that starts with a slash is read as an app command.

Frequently asked questions