In 2024, a Canadian tribunal ordered Air Canada to compensate a passenger after the chatbot on its website described a bereavement-fare refund rule that didn’t exist. The airline argued that the chatbot was responsible for its own words. The tribunal disagreed. A year earlier, two New York lawyers were sanctioned for a court filing that cited cases ChatGPT had invented. In 2025, Deloitte refunded part of its fee to the Australian government after a report it delivered turned out to contain fabricated references.
Your stakes are probably smaller, but the pattern is the same. A supplier invoice gets booked with a due date that wasn’t on the PDF. A customer is told an item is in stock when it isn’t. A manager acts on a number nobody checked. You can’t make a language model that is never wrong. You can design a system in which wrong answers get caught before they cost you anything, and this article shows how.
What a hallucination is
A hallucination is an answer that sounds confident and correct but isn’t supported by facts. It isn’t a bug that a software update will remove. It’s a side effect of how language models write. Common forms:
- Invented facts: a refund policy, a product specification, a paragraph of a law that doesn’t exist.
- Fake sources: citations, links, case numbers or report titles that look real and aren’t.
- Wrong numbers: totals, dates, percentages, a VAT amount that doesn’t add up.
- Misreading the source: the contract says 30 days, the summary says 60.
- Filling in blanks: asked for the due date on an invoice that has none, the model supplies a plausible one.
The last two matter most in business, because they happen even when you gave the model the right document.
Why it happens
A language model writes by predicting likely text, one piece at a time (we explain this in what generative AI is). Nothing in that process checks the text against reality. The specific causes:
- A question without the facts. If your price list, policy or yesterday’s data isn’t in front of the model, it can only produce what such an answer usually looks like.
- Guessing gets rewarded. In a 2025 paper, researchers at OpenAI argued that the way models are trained and scored rewards a confident guess over “I don’t know”, much like an exam where a blank answer scores zero. So models learn to guess.
- Tasks the format is bad at: arithmetic, counting, exact quotes, dates.
- Buried details. In a long document, the one line that matters is easy to miss.
- Leading questions. “Why is our contract with this supplier invalid?” invites a story about why it is invalid.
When it’s dangerous, and when it isn’t
Not every wrong sentence matters. The risk is the cost of an error multiplied by the chance that nobody notices before it does damage.
| Risk | Examples | What it needs |
|---|---|---|
| Low | Brainstorming, first drafts a person rewrites, notes for someone who was in the meeting | A quick read |
| Medium | Customer replies an agent reviews, invoice fields checked by rules | Sources, validation, review of exceptions |
| High | Anything sent unchecked; legal, tax, medical or financial statements; prices and commitments to customers; figures management will act on | All of the techniques below, and a person signs off |
Match the effort to the risk. A team that double-checks every brainstorming list wastes its time. A team that lets a chatbot quote prices unchecked is taking a real risk.
Seven ways to design around it
1. Give it the source
This is the biggest single improvement: don’t ask the model to know things. Find the relevant documents or records, give them to the model together with the question, and tell it to answer only from them. The technique is called RAG, and our plain-English explainer shows how the search part works. For live data, let the model query the database directly. That’s how we built Charlie for Shovels: it answers questions about construction permits and contractors from current database records, not from whatever the model happens to remember.
Same question, two designs
Answer from memory
- Customer asks about returning an opened item
- Model writes a typical policy: 30 days
- No source to check against
- A promise you never made
Answer from your documents
- Search finds your returns policy, section 4
- Model answers: unused items, within 14 days
- The answer links to section 4
- Nothing relevant found? It says so and hands over
2. Ask for citations, then check them in code
Ask the model to name the document and quote the exact sentence its answer relies on. Then let your software check that the quote really appears in that source. If it doesn’t, the answer is flagged for a person. This is cheap to build, and it catches many invented details, because a made-up claim rarely comes with a real quote.
3. Let it say “I don’t know”
Left to itself, the model prefers to answer. Give it an explicit, acceptable way out: “If the documents don’t answer the question, say you don’t have that information and offer to pass it to a colleague.” Then build the hand-off, so that “I don’t know” leads somewhere useful instead of a dead end.
- You: Can I return an opened coffee machine after 10 days?
- Tool call: search_help_centre("return opened appliance")
- Result: Returns policy, section 4: unused items within 14 days. Nothing about opened appliances.
- AI: Our policy covers unused items returned within 14 days. It doesn’t say anything about opened appliances, so I don’t want to guess. Shall I pass this to a colleague?
- You: Yes, please.
- Handed over with the chat and the policy section
- Peter: Hi, Peter here. I can see your question. I’ll check with our service team and get back to you today.
4. Fix the format and validate it
For extraction, don’t ask for free text. Ask for a fixed set of fields (developers call this structured output) and allow each field to stay empty when the document doesn’t contain it. Then check the result with ordinary rules.
An accounting firm processing supplier invoices might check that the line items add up to the total, the VAT matches the rate, the due date falls after the issue date, the supplier and their IBAN already exist in the system, and the invoice number isn’t a duplicate. Anything that fails goes to a person.
Invoice extraction with checks
- Invoice PDF arrives
- AI extracts fields Fixed fields, left blank if missing
- Rules check Totals, VAT, dates, supplier, duplicates
- Person reviews flags Only invoices that failed a check
- Booked in accounting
5. Use tools for facts and maths
Don’t let the model produce numbers it could look up or calculate. Give it tools instead: a stock lookup, a price query, a calculator, a calendar. The model decides what to look up and writes the sentence around the result, but the number itself comes from your system. A distributor’s sales assistant should never “remember” a price. It should fetch it.
6. Put a person where the risk is
Decide in advance which outputs a person checks. At the start, check everything. As the error rate drops, move to checking only exceptions plus a random sample. Anything irreversible, such as paying money, deleting data or making a commitment to a customer, needs an approval step. We cover these patterns in human in the loop.
7. Test with real examples, every time something changes
Collect 50–100 real cases with known correct answers: typical ones, tricky ones, and questions your documents can’t answer. Run them whenever you change the prompt, the documents or the model, and track how many come out wrong. Without such a test set, you learn about errors from your customers. Building one is part of what separates a pilot from a production system.
Putting it together: a clinic’s front desk
A private clinic wants an assistant that answers patients’ messages about appointments and how to prepare for an examination. This is how the techniques combine:
- The assistant answers only from the clinic’s own preparation instructions, price list and opening hours, which the staff keep up to date (technique 1).
- Every answer links to the instruction it used, and the system checks that the quoted lines exist (2).
- Questions about symptoms, results or medication get a fixed reply and go to a nurse. The assistant never improvises medical advice (3 and 6).
- Free slots and prices come from the booking system, not from the model (5).
- Before launch, the staff collect 80 real patient messages, awkward ones included, and the team reruns them after every change (7).
Because patient messages contain health data, the clinic also decides where the model runs and what gets stored. The result is less impressive in a demo than a chatbot that answers everything. It is much safer in daily use, and the nurses trust it, which in the end decides whether anyone uses it.
What doesn’t work
- “Don’t make things up” in the prompt. It may help a little, but it isn’t a control. The model can’t reliably tell which of its own sentences are unsupported.
- Trusting the tone. A confident, well-formatted answer is no more likely to be right.
- Waiting for a better model. Newer models make fewer mistakes, but not zero. Your process still has to catch the rest.
- Setting the temperature to zero. Temperature is a setting that controls how random the output is. At zero, answers become more repeatable, not more correct. A wrong answer simply becomes a consistently wrong answer.
- Asking “are you sure?” The same model may defend a wrong answer or drop a right one. A second check only helps when it compares the answer against the source.
Checklist before you go live
Before an AI feature goes live
- The model answers from sources you supply, not from memory
- Every answer points to its source A document, a section or a record
- Quotes and numbers are checked in code Does the quote exist? Do the totals add up?
- It can say “I don’t know” And the hand-off to a person works
- Extraction uses fixed fields that may stay empty
- Prices, stock and maths come from tools, not the model
- A person approves anything risky or irreversible
- A test set of real cases Rerun after every change to the prompt, documents or model
Most of these safeguards are ordinary software around the model, and that’s where we spend most of our time when we build AI features for clients. If you have a process where a wrong answer would be expensive, let’s talk it through.
