AI hallucination

An AI hallucination is a statement a language model produces with normal fluency and normal confidence that is not true — a fabricated citation, an invented price, a policy that no one wrote. The defining property is not the error; it is that nothing in the output marks it as different from the parts that are correct.

Where it comes from

Hallucination is usually described as a mystery. The clearest published account says the opposite. In Why Language Models Hallucinate, submitted on 4 September 2025, Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang argue that hallucinations "originate simply as errors in binary classification": if incorrect statements cannot be distinguished from facts, then hallucinations in pretrained models "will arise through natural statistical pressures".

Their second argument is the one that matters for anyone reading model output. Hallucinations persist not because pretraining is hard to fix but because of how evaluations are graded. Models are "optimized to be good test-takers, and guessing when uncertain improves test performance". A benchmark that scores an abstention exactly as it scores a wrong answer has priced honesty at zero. The authors call the fix socio-technical: change the scoring of the benchmarks that dominate leaderboards, rather than adding more hallucination evaluations alongside them.

Two consequences follow that are easy to miss. First, a hallucination rate is a property of an objective, not a defect in one vendor's model. Second, the failure is systematically worst exactly where the training distribution is thin — obscure entities, recent events, small brands — because that is where guessing beats abstaining by the widest margin.

The two kinds, and why the difference matters

They are not one phenomenon.

• Parametric hallucination. The model answers from weights alone. There is no document behind the claim, no fetch, and no URL to inspect. Its content is a compressed impression of training data fixed months earlier.
• Grounded hallucination. The model was given documents and still wrote something they do not support. It misread a passage, merged two, or extended a qualified claim past its qualifier.

Only the second is visible from outside, and only partially. In a RAG system a citation records what was retrieved, not that the retrieved passage supported the sentence attached to it. A confidently wrong answer with a real, working link to your page is the normal shape of this failure, and it is the shape that damages a publisher, because the link makes the claim look checked.

Why a brand should care about the mechanism, not just the rate

Because the two kinds have different remedies, and only one of them is available to you.

You cannot edit a model's weights. You can change what a retrieval step finds. That makes the practical question narrow: when an agent fetches a page about your product, is the correct claim present, unambiguous, and adjacent to its own qualifier?

Most brand-level hallucinations traced back to a source are not inventions at all. They are correct sentences read out of context: a launch price still sitting on a 2023 announcement page, a limit described in a paragraph whose "on the legacy plan" caveat sits two paragraphs above it, a feature name reused for two different things. The model did not make the claim up. It extracted it faithfully from text that was true only in a context it did not carry.

Failure modes to look for

1. The orphaned qualifier. A number is right; the condition on it is in another paragraph. Extraction takes the number and leaves the condition.
2. The stale twin. Two pages state a fact, one current, one from an old release. Both are fetchable, and nothing tells a model which is authoritative.
3. The invented identifier. Version numbers, SKUs and API names are exactly where guessing is most likely and hardest for a reader to spot. Anthropic's own guidance suggests extracting word-for-word quotes first for long documents, precisely because paraphrase is where identifiers drift.
4. The plausible source. A fabricated reference that has a real-looking title and a dead or unrelated URL. This is the failure the source citation chain exists to make detectable.
5. The silent gap. Nothing about you is retrievable, so the model answers from a thin parametric impression. Absence does not produce silence; it produces a guess.

What actually reduces it

Grounding, and refusal permission. Anthropic's published guidance on reducing hallucinations is short and blunt: allow the model to say "I don't know", ground responses in direct quotes rather than paraphrase, and have it verify each claim against a supporting quote — retracting any claim it cannot support.

On the publishing side the equivalent moves are structural, not persuasive. Write each claim so it survives being lifted alone. Keep one current statement of each fact, and remove or date the superseded ones. Keep identifiers as literal text. None of that makes a model honest; it removes the ambiguity that a guessing model resolves badly.

Frequently asked questions

Why guess instead of abstaining?

Because standard grading scores an abstention like a wrong answer, so guessing is the better strategy.

Does retrieval fix it?

No. It changes where the text comes from, not whether the model reads it correctly.

Is a citation proof?

No. A citation records what was retrieved, not that it supported the claim.

What is in a publisher's control?

The retrievable text: one current version of each claim, qualifiers in the same sentence, identifiers written literally.