Why AI makes things up — a hallucination caught in the act

"Hallucination" is explained badly almost everywhere, usually with some version of "it just predicts the next token" — true, and completely unsatisfying. What I had instead was an unusually clean specimen: a single wrong token in an otherwise perfect answer, in a test where I could check whether the model actually knew better. It did. That gap between knowing and writing is the whole story.

The setup

I was benchmarking our local coding model by executing its output against hidden tests it had never seen — not reading the code and nodding, actually running it. The hard set was six problems built to resist memorisation: multi-unit duration parsing, greedy weight-packing with an oversized-item edge case, a three-way dictionary diff, Roman numerals with subtractive pairs, an incremental running median, and a bracket validator that also has to respect quoted strings.

Twelve runs, two temperatures each. Every single one compiled. Ten of twelve produced correct output. The two failures were the same problem, and the same mistake, at both temperatures.

The mistake

The running-median problem wants the classic two-heap solution: a max-heap for the low half, a min-heap for the high half, rebalanced as values arrive. The model got all of that right. The algorithm was correct. The even/odd averaging at the median boundary was correct. The docstring was correct.

Then the import line said:

from heapq import heappush, heappoppop

There is no heappoppop in Python. The function is heappushpop. The program died at import with ImportError: cannot import name 'heappoppop', having never executed a single line of the correct algorithm sitting underneath it.

Look at the shape of that error. It isn't a wild invention. It's a real function name with its two halves scrambled — push and pop are both genuinely in there, in the wrong arrangement. That's what most hallucinations look like up close: something adjacent to the truth, assembled from the right parts.

The part that makes it interesting

Failing identically at both temperatures is suspicious. When a model produces the same error regardless of sampling randomness, my first instinct is that my own harness is broken — I've been burned by that enough times to check reflexively.

So I asked the model directly, in plain conversation: what does heapq export? It listed heappushpop, correctly, with an accurate description of what it does. It never produced heappoppop. Asked again a different way, same result.

That answer travelled through the same rendering path as the broken code, so the app wasn't corrupting anything. The model knew. Ask it as a question and the knowledge is right there. Ask it to write forty lines of correct heap logic and somewhere in the middle of that, generating the import line almost as an afterthought, it fumbled the token.

I find this genuinely clarifying. The model isn't missing a fact. It's failing to retrieve a fact precisely while doing something else demanding. If you want a human analogy — and analogies here are always a bit loose — it's closer to a typo than to ignorance.

Why "it looked right" is not a check

Here's the practical sting. That file compiled perfectly.

Python's compiler checks syntax. It does not check that the names you're importing exist — that happens later, at import time. So any workflow that gates AI-generated code on "does it compile" would have waved this straight through. In my evaluation, compile-checking would have caught exactly zero of the one real failure in the entire suite.

This generalises well beyond code. A hallucinated citation is formatted like a citation. A hallucinated statistic has the right number of digits. A hallucinated API is plausibly named. The output passes every surface check precisely because surface plausibility is what the model optimises. Fluency is not evidence. The only checks that catch this class are ones that touch reality: run the code, open the source, verify the symbol exists.

Where this shows up for you

The pattern is constant: the more confident and well-formed the output, the less that confidence tells you. Models don't signal uncertainty where they're wrong, because they aren't tracking truth in the first place.

Does running it locally change any of this?

No, and I'd rather say so plainly than imply otherwise. Hallucination is a property of how language models work, not of where they run. A model on your Mac invents things for exactly the same reasons a cloud model does.

What local execution changes is elsewhere — privacy, cost, whether your work stops when your connection does. On factual reliability, treat every model the same way and verify anything load-bearing. Bigger models slip less often, but "less often" is not "never", and the failure looks identical when it happens.

What we're doing about this specific class

For the API-name case there's a cheap, targeted fix: resolve symbols against the standard library before handing code back. Ask whether heapq genuinely exports heappoppop, and you catch this instantly — the standard library is identical on every Python installation, so the check is safe and fast.

What we deliberately don't do is check a user's own project imports the same way. Judging whether some package exists by inspecting the app's own environment produces confident, wrong answers about someone else's setup, which is just a different hallucination with our name on it.

And it's worth keeping the failure in proportion: across that hard novel set, 12 of 12 compiled and 10 of 12 ran correctly — 83% on problems built specifically to defeat memorisation, with the only genuine failure being this one name. On everyday tasks the same evaluation ran at effectively 100%. The gap is precision-of-recall, not reasoning.