What is an AI hallucination?

What an AI hallucination is

An AI hallucination is a confidently stated output from a generative model that is not grounded in fact or in the provided source material. The term covers everything from a wrong date or a misattributed quote to a wholly invented statistic, study, or legal citation. IBM describes hallucination as occurring when a model perceives patterns that do not exist and produces nonsensical or inaccurate output; a 2023 survey by Huang and colleagues defines it succinctly as "generating plausible yet nonfactual content."

The defining trait, and the source of the risk, is that hallucinated output is fluent and self-assured. The model does not signal uncertainty or flag that it is guessing. To a reader, a fabricated answer is indistinguishable from a correct one on the surface, which is why hallucination is especially dangerous in domains where accuracy matters.

Why models hallucinate

The cause is not a bug but a consequence of how large language models work. An LLM generates text by predicting the most statistically probable next token given the preceding context. It is optimizing for plausibility, output that resembles its training data, not for truth. The model has no internal database of verified facts and no built-in mechanism to check a claim against reality before producing it.

Several factors make it worse. Training data may be incomplete, outdated, biased, or contain errors that the model reproduces. When asked about something rare or absent from its training, the model still generates a confident answer by pattern-matching, effectively filling the gap with a statistically likely fabrication. Ambiguous prompts, and the model's general tendency to be helpful and produce an answer rather than decline, also increase the risk.

Because the mechanism that makes LLMs fluent and creative is the same one that lets them hallucinate, the two cannot be cleanly separated. This is why hallucination is treated as an inherent limitation of current architectures rather than a defect to be patched away.

Types of hallucination

Researchers commonly split hallucinations into two broad categories. Factual hallucinations are outputs that contradict verifiable real-world facts, for example, stating the wrong founding year of a company or inventing a scientific finding. These are failures against external truth.

Faithfulness (or intrinsic) hallucinations are outputs that contradict, or are unsupported by, the specific source material or instructions the user provided. In a summarization task, a model that adds a detail not present in the source document has produced a faithfulness hallucination even if the added detail happens to be true elsewhere. This distinction matters because grounding a model in a document reduces factual errors but does not automatically guarantee faithfulness.

The 2023 survey by Huang et al. formalizes this taxonomy for LLMs and analyzes the underlying causes, making the factual-versus-faithfulness split a standard lens for evaluating and mitigating the problem.

Real-world risks

Hallucination becomes a serious liability in YMYL ("your money or your life") domains, finance, medicine, law, and safety, where a confident false answer can cause direct harm. A fabricated drug interaction, an invented tax rule, or a nonexistent legal precedent can lead a user badly astray precisely because it reads as authoritative.

The most cited real-world example is the 2023 U.S. federal case Mata v. Avianca. Lawyers filed a brief that cited multiple prior cases which did not exist; the citations had been generated by ChatGPT, complete with realistic-looking quotations. When neither opposing counsel nor the court could locate the cases, Judge P. Kevin Castel of the Southern District of New York sanctioned the attorneys and their firm, imposing a USD 5,000 penalty on 22 June 2023.

Beyond individual harm, hallucination erodes trust in AI systems and can pollute the information ecosystem, as false claims get quoted, indexed, and repeated. For any organization publishing AI-assisted content, an unverified hallucination is a reputational and, in regulated areas, a legal risk.

How hallucination is mitigated

No single fix solves hallucination, so mitigation stacks several techniques. Retrieval-augmented generation (RAG) is among the most effective: relevant, verified documents are retrieved and inserted into the model's context, and the model is instructed to answer only from that material and cite it. Grounding output in a trusted knowledge source sharply reduces factual invention.

Reinforcement learning from human feedback (RLHF) trains models to prefer accurate, honest, and appropriately hedged responses, including saying "I don't know." Prompt engineering helps too: clear instructions, supplied context, and explicit permission to decline reduce the pressure on the model to fabricate. Google Cloud and IBM both emphasize grounding, retrieval, and constraining the model to reliable sources as core mitigations.

The final and non-negotiable layer is verification, automated fact-checking against primary sources and human review before publication. For high-stakes output, treating every model claim as a draft to be checked, rather than a finished fact, is the only responsible workflow.

Why it cannot be fully eliminated

It is important to be honest: with current technology, hallucination can be reduced but not eliminated. The behavior stems from the fundamental design of generative models, they produce statistically likely text rather than retrieving verified truth, so as long as a model is generating rather than looking up, some probability of fabrication remains.

Even a well-grounded RAG system can misread its retrieved sources, blend them incorrectly, or hallucinate in the gaps between them. Mitigations lower the rate and the severity, and they enable citation and traceability, but they do not turn a probabilistic generator into an oracle. Vendors and researchers consistently frame residual hallucination as a known, ongoing limitation.

The practical takeaway is a design principle, not a promise: build systems that assume hallucination is possible, ground answers in verifiable sources, cite them, and keep a human in the loop for anything that matters. Trustworthy AI is achieved through verification, not by pretending the risk has been removed.

Frequently asked questions

Why do large language models hallucinate?

Because they generate text by predicting the statistically most likely next token, optimizing for plausibility rather than truth. An LLM has no internal store of verified facts and no built-in way to check a claim before producing it, so when information is missing, ambiguous, or absent from training, it fills the gap with a confident, plausible-sounding fabrication. The same generative mechanism that makes models fluent is what makes them hallucinate.

Can AI hallucinations be eliminated?

No. With today's technology they can be substantially reduced but not fully eliminated. Techniques like retrieval-augmented generation, grounding, RLHF, and human verification lower the rate and severity, but because generative models produce probable text rather than retrieving verified truth, some risk of fabrication always remains. Responsible systems assume hallucination is possible and rely on grounding, citation, and human review.

What is the difference between factual and faithfulness hallucinations?

A factual hallucination contradicts verifiable real-world facts, such as inventing a statistic or a study. A faithfulness hallucination contradicts or goes beyond the specific source material or instructions the user provided, for example, adding a claim to a summary that is not in the source document. Grounding a model in a document reduces factual errors but does not automatically guarantee faithfulness.

Has an AI hallucination caused real harm?

Yes. In the 2023 U.S. case Mata v. Avianca, attorneys submitted a legal brief citing several prior cases that did not exist; the citations had been generated by ChatGPT. Judge P. Kevin Castel of the Southern District of New York sanctioned the lawyers and their firm with a USD 5,000 penalty on 22 June 2023. The case is now a standard example of hallucination causing professional and legal consequences.

Does retrieval-augmented generation (RAG) stop hallucinations?

RAG reduces hallucination but does not stop it entirely. By retrieving verified documents and instructing the model to answer only from them and cite them, RAG grounds responses in real sources and cuts factual invention significantly. However, a RAG system can still misread its sources, combine them wrongly, or hallucinate in the gaps between them, so verification remains necessary.

How can users reduce the chance of getting a hallucinated answer?

Provide clear, specific prompts with relevant context; ask the model to cite sources and to say when it does not know; prefer tools that ground answers in retrieved, verified documents; and independently verify any high-stakes claim, especially figures, dates, quotes, and citations, against a primary source before relying on it. Treat model output as a draft to be checked rather than a finished fact.

Related

Explainers: LLM, RAG, Prompt engineering

Glossary: Hallucination, RAG (retrieval-augmented generation), RLHF (reinforcement learning from human feedback), Large language model (LLM), Prompt engineering

Sources

• Huang et al. (2023) — A Survey on Hallucination in Large Language Models
• IBM Think — What are AI hallucinations?
• Google Cloud — What are AI hallucinations?
• Mata v. Avianca, Inc. (S.D.N.Y. 2023) — case reference