An AI hallucination is an answer from a language model that sounds confident and plausible but does not match reality: an invented fact, a non-existent source, a wrong figure. The term is a metaphor, since a model has no perception that could deceive it. Some researchers prefer "confabulation", but "hallucination" is the word that stuck.
The cause lies in how the model works. It builds an answer as the most probable continuation of the text and by default does not check it against a source. When the required knowledge is missing, a probable continuation is still found, and it reads just as smoothly as a correct one. A model can say "I don't know", but it does so less often than it should unless asked to.
Most often the model invents something that resembles the real thing: a reference to a legal provision with a plausible number, a quotation, the title of a study, a function in a software library, a clause in a contract. It can be subtler. The model receives a real document but, when summarising it, mixes up a date, attributes a condition to the wrong party or adds a clause that is not there.
The danger is not the error itself, since people make mistakes too. The danger is that a wrong answer sounds exactly like a right one, and the only way to catch it is to check it against the source.
In 2023, lawyers in New York filed a court brief citing rulings generated by ChatGPT. Several of those cases did not exist. The court fined the lawyers, and the story became a textbook example.
In 2024, a chatbot on Air Canada's website told a passenger about refund terms that were not in the airline's rules. A Canadian tribunal ordered the company to pay the difference: a company is responsible for what its bot says on its website. For a business this is the key lesson: responsibility for an AI answer lies with whoever put it in front of the customer.
Each generation of models hallucinates less, yet the problem does not disappear completely: it is the flip side of the model's ability to phrase things freely. So the question is framed differently: where an error is acceptable, where a check will catch it, and where the cost of an error is so high that no answer may go out without a human review.
The most effective measure is to give the model a source and ask it to answer only from that source. For company documents this means search over them (RAG) with a link to the document in every answer. A person can open the link and check.
It also helps to explicitly allow the answer "this is not in the documents" and to describe in the prompt or specification what to do when data is missing. Where an answer feeds into a business system, code checks it: format, permitted values, agreement with the source data. Where money or commitments to a customer are at stake, the decision stays with a human in the loop.
In a catalogue automation project for an online shop, AI wrote descriptions and keywords for products sold in several colours. Reviews of successive versions kept turning up the same errors: product keywords included colours the product did not come in, banned marketing phrases appeared, languages got mixed. Each text looked tidy, and on its own the error was easy to miss.
Manual fixes in finished texts did not hold: the error came back in the next run. Things started working once the fixes moved into the generation code and automatic checks were added to reject a text before export. More about the project in the portfolio.
Where in the process an AI answer moves on without a person, and what an error at that step would cost. Which sources the model takes its facts from and what it answers when they contain nothing. Which checks code performs and which a person performs. And which set of test questions with known answers goes into the acceptance criteria, so the error rate is measured before launch.
Related terms: RAG · Human in the loop · Acceptance criteria · Prompt
Want to know where an AI error in your process would cost the most? AI audit for a department →