Working with AI
What is AI hallucination?
An AI hallucination is when a language model states something false with full confidence: an invented statistic, a non-existent policy, a fabricated citation, produced because the model predicts plausible text rather than checking facts.
Hallucination is not a bug that better models will simply patch away; it is a direct consequence of how LLMs work. The model produces the most plausible continuation of the text. Usually the plausible thing is true. Sometimes the plausible thing is a confident invention, and nothing in the raw model marks the difference. That asymmetry (fluent delivery, unverified content) is why unguarded chatbots occasionally promise customers refund policies that do not exist.
The fix is architectural, not motivational. Production systems reduce hallucination by grounding answers in retrieved documents (RAG) and instructing the model to answer only from them, by validating structured outputs against the real database (does that order number exist?), by giving the system an explicit "I don’t know, let me hand you to a human" path, and by logging conversations so the remaining failures get found and fixed. On sensitive flows, a second model can check the first one’s claims before anything reaches the customer.
In our own builds this is standing policy: the assistant is allowed to say it does not know, and honesty beats fluency wherever the data is missing. Tamr’s AI waiter answers allergen questions only from what the restaurant actually provided, because a smooth guess about allergens is not a wrong answer, it is a dangerous one.
Buyer’s test: ask a vendor "what does your system do when it doesn’t know?" A real answer describes retrieval, validation, and handover. A hand-wave means the failure mode is yours to discover in production.
Frequently asked
- Why do AI models hallucinate?
- Because they predict the most plausible continuation of text, not the true one. Usually plausible and true line up; sometimes the plausible thing is a confident invention, and nothing in the raw model marks the difference. It is a direct consequence of how LLMs work, not a bug that a better model simply patches away.
- How do you stop AI from hallucinating?
- The fix is architectural, not motivational. Ground answers in retrieved documents and instruct the model to answer only from them, validate structured output against your real database, give the system an explicit "I don’t know, let me hand you to a human" path, and log conversations so remaining failures get found. On sensitive flows, a second model can check the first one’s claims before anything reaches the customer.
- How do I test a vendor on hallucination?
- Ask them what their system does when it does not know. A real answer describes retrieval, validation, and handover. A hand-wave means the failure mode is yours to discover in production. Our own standing policy is that the assistant is allowed to say it does not know, because a smooth guess about something like allergens is not a wrong answer, it is a dangerous one.
Wondering where this fits in your business?
The free AI audit maps your operations and shows the three systems we would build first, in plain language like this.
