AI Hallucinations: Why AI Lies Confidently and What to Do About It
AI hallucinations are not a bug being patched out — they are a structural property of how language models work. This matters enormously if you use AI for research, legal work, medical questions, or any fact-based writing.
A lawyer in New York submitted a legal brief citing six precedent cases. None of them existed. ChatGPT had invented them — case names, docket numbers, quotes from judges, page citations — with the same confident, fluent prose it uses to summarize real cases. The lawyer, who hadn't verified the citations, was sanctioned by a federal judge, publicly reprimanded, and fined $5,000. The cases had never happened. The court had never ruled. The AI had simply generated text that looked exactly like what a real legal citation looks like, because that is what it was trained to do.
This is not an isolated story. It is a structural feature of how every major AI language model works — and it will not be fixed by the next model release, or the one after that. Understanding why hallucinations happen is the first step toward knowing when to trust AI output and when verification is not optional.
~20%
hallucination rate in AI-generated text summaries in controlled studies
Ji et al., ACM 2023 [1]
69%
of ChatGPT-generated medical paper references contained significant errors
Alkaissi & McFarlane, 2023 [2]
$5K
fine issued to lawyer who submitted AI-fabricated citations in federal court
Mata v. Avianca, 2023 [3]
$100B
market cap erased from Google after Bard hallucinated in its first public demo
Reuters, Feb 2023 [4]
What Is a Hallucination, Exactly?
The term "hallucination" in AI refers to outputs that are factually incorrect, fabricated, or unsupported — but presented with the same confidence and fluency as accurate outputs. The word is borrowed from psychology, where it means perceiving something that isn't there. In AI, the analogy is apt: the model generates text that sounds completely real and accurate while describing something that simply isn't.
What makes AI hallucinations uniquely dangerous compared to other kinds of errors is that there is no signal. When a calculator divides by zero, it returns an error. When a search engine can't find something, it says so. When an AI halluccinates, it returns a confident, well-formatted, grammatically perfect answer. The error is invisible in the output itself.
Why This Is Structural, Not a Bug
Most people assume hallucinations are a temporary engineering problem that will be fixed as models improve. This reflects a misunderstanding of how language models work. Hallucinations are not a bug; they are an emergent property of the architecture itself.
How Language Models Actually Generate Text
1.
Token prediction, not knowledge retrieval. When you ask an AI a question, it does not look up the answer in a database. It predicts, one token at a time, what text is statistically most likely to follow your input — based on patterns in billions of documents it was trained on. It has no separate "fact store" to check against.
2.
Confidence comes from fluency, not accuracy. The model is trained to generate coherent, fluent text. Confidence is a stylistic output — the result of training on text where assertions are typically made in confident prose. The model "sounds sure" because its training data was full of text where assertions sound sure. There is no internal mechanism that calibrates expressed confidence to actual accuracy.
3.
The model fills gaps plausibly, not truthfully. When asked about something it has incomplete or no training data on — a niche legal case, an obscure paper, a specific recent event — the model continues generating text anyway, producing whatever completion is statistically plausible given the surrounding context. For a legal citation, that means: a case name that sounds like a case name, a docket number that looks like a docket number, a holding that sounds legally coherent. All fabricated.
4.
RLHF and fine-tuning reduce but do not eliminate this. Techniques like reinforcement learning from human feedback (RLHF) train models to be more accurate and to express uncertainty when appropriate. This measurably reduces hallucination rates on common questions. It does not solve the underlying architecture. The model is still predicting tokens, not retrieving verified facts.
Researchers have documented that hallucination rates are higher for less popular entities — obscure people, niche topics, specific figures — because the training data contains less information to pattern-match against, so the model fills more gaps with plausible-sounding fabrications. A 2023 study by Mallen et al. found that LLMs answered questions about popular entities like major historical figures significantly more accurately than questions about less-commonly-discussed subjects, where hallucination rates spiked.
When Hallucinations Caused Real Harm
These are not hypothetical risks. Each of the following cases involves AI-generated hallucinations that led to documented real-world consequences.
⚖️
Mata v. Avianca — The Fake Legal Brief (2023)
US District Court, SDNY · Judge P. Kevin Castel
Attorney Steven Schwartz used ChatGPT to research a personal injury case against Avianca airline. ChatGPT produced a brief citing six precedent cases: Varghese v. China Southern Airlines, Shaboon v. Egyptair, Petersen v. Iran Air, Miller v. United Airlines, Estate of Durden v. KLM Royal Dutch Airlines, and Zicherman v. Korean Airlines. When Avianca's attorneys attempted to locate these cases, none could be found. When the court ordered Schwartz to produce the actual opinions, ChatGPT doubled down and "confirmed" the cases were real. They were not. They had never existed.
Judge Castel found that Schwartz had failed to perform basic verification of his sources. The court imposed $5,000 in sanctions and ordered Schwartz to notify the judges cited in the fictional opinions. The case became one of the most widely cited examples of AI hallucination harm in professional practice.
Outcome$5,000 fine, public sanctions, mandatory notifications to cited judges. No fictional cases were ever real.
🔭
Google Bard's $100 Billion Hallucination (February 2023)
Google / Alphabet · Launch demo · Reuters fact-check
In a promotional GIF released by Google ahead of Bard's launch, the AI was asked what new discoveries the James Webb Space Telescope had made that could be shared with a 9-year-old. Bard answered that JWST "took the very first pictures of a planet outside of our own solar system." This was incorrect — the first images of an exoplanet were taken in 2004 by the Very Large Telescope at the European Southern Observatory.
Reuters flagged the error before Bard was even publicly available. Alphabet's stock fell approximately 8% the following day, wiping roughly $100 billion from its market capitalization — a pointed demonstration that hallucinations carry real financial consequences even before a product launches.
Outcome~$100B market cap loss in a single trading session. The error occurred in a controlled promotional environment, not in unguided user interaction.
✈️
Air Canada Chatbot — Legally Binding Hallucination (2024)
British Columbia Civil Resolution Tribunal · February 2024
Jake Moffatt contacted Air Canada's customer service chatbot after his grandmother died to ask about bereavement fares. The chatbot told him he could book a ticket at full price and then apply for a bereavement discount retroactively within 90 days. This was incorrect — Air Canada's actual policy required the discount to be requested before travel. Moffatt booked the ticket based on the chatbot's instructions, paid full price, and then was denied the discount when he applied.
Air Canada argued in the tribunal that the chatbot was "a separate legal entity" responsible for its own statements, and that the airline was not responsible for its chatbot's errors. The British Columbia Civil Resolution Tribunal rejected this argument entirely, ruling that Air Canada is responsible for all information on its website regardless of whether it comes from a static page or an AI chatbot. Air Canada was ordered to pay Moffatt the difference in fare plus interest and fees.
OutcomeAir Canada held legally liable for chatbot hallucination. "Separate legal entity" defense explicitly rejected. Binding precedent for corporate AI liability.
🏥
Medical Citation Fabrication at Scale (2023)
Alkaissi & McFarlane · Cureus Medical Journal
Researchers Hussam Alkaissi and Safi McFarlane asked ChatGPT to generate a list of references for a medical article on mesh implants. Of the references returned, 69% contained significant errors: fabricated authors, incorrect titles, journals that didn't exist or didn't publish the cited papers, wrong volume and page numbers. Several citations appeared real enough to pass a casual check — the journal names, author name formats, and publication structures all conformed to real academic citation conventions.
The researchers noted that the errors were not random — they were systematically plausible, constructed to look like real medical literature. This is the defining feature of AI hallucinations: they are not obviously wrong. They require active, deliberate verification to catch.
Finding69% error rate in AI-generated medical citations. Errors were specifically designed (by statistical training) to appear plausible and real.
What a Hallucination Looks Like in Practice
The most dangerous hallucinations are not obviously wrong. They are wrong in ways that require domain expertise and active verification to catch. Here is a real pattern of hallucination output — the kind that passes a surface read:
⚠ AI Output Example — Contains Fabrications
"A 2021 meta-analysis by Hendricks et al. published in the Journal of Digital Health examined 14 randomized controlled trials (n=6,240) and found that AI-assisted diagnostic tools reduced misdiagnosis rates by 34% (95% CI: 28–41%) compared to unaided clinical assessment. The authors noted significant heterogeneity across surgical specialties (I²=67%)."
↑ Underlined elements are fabricated. "Hendricks et al." does not exist. The journal may exist but published no such paper. The sample size, confidence interval, and heterogeneity figure are invented. Every formatting detail is statistically plausible — which is why this passes a surface read.
⚠
The most important thing to understand about hallucinations: the model does not know it is wrong. It is not lying. There is no deliberate deception. The model is doing exactly what it is designed to do — generate statistically plausible text — and the result happens to be false. This means there is no "tell" in the output itself. Hallucinations and accurate outputs are stylistically identical.
What to Never Trust AI With (And What's Safer)
The risk of hallucination is not uniform across all AI use cases. The critical distinction is whether the task requires factual accuracy that can be independently verified — and whether the consequences of an error are significant.
✗ Never Trust Without Verification
Specific legal citations, case names, docket numbers
Medical studies, clinical trial results, dosages, drug interactions
Brainstorming and generating options where you evaluate
Explaining concepts from its training (broad, well-documented)
Creative writing, fiction, hypothetical scenarios
Code generation (verify it runs; logic errors are different from hallucinations)
Translation of text you provide into another language
Sentiment analysis and classification of provided text
→
The distinction between "AI working from your source" and "AI generating facts from memory" is the most important practical dividing line. Google NotebookLM (free) lets you upload documents and ask questions against them — dramatically reducing hallucination risk for research tasks. See our AI tools guide for more on safe research workflows.
The Verification Workflow
Knowing hallucinations exist is not enough. You need a practical habit for the tasks where hallucinations cause the most damage. This checklist applies any time you use AI output that contains specific factual claims.
✓
Before Using AI-Generated Factual Claims
Apply to: research, legal work, medical questions, journalism, financial analysis
1
Identify every specific factual claim in the output.
Names, dates, numbers, citations, statistics, and attributions are all specific claims. Highlight them before reading the surrounding prose, which can make the claims feel more credible than they are.
2
Verify citations at the primary source, not via another AI.
Asking a second AI to verify a citation from the first AI will often produce confirmation, not correction — the second model draws on the same training distribution. Verify legal citations on Westlaw/LexisNexis, academic citations on PubMed/Google Scholar, news claims on the original publication.
3
Be more suspicious of obscure, specific, or niche claims.
Research shows hallucination rates spike for less-documented topics. If AI cites a specific subsection of a regulation, a minor paper, or an unusual historical detail, verify it. Well-known general facts are more reliable than specific obscure ones.
4
Ask the AI to express its uncertainty — then verify the things it's confident about most carefully.
Prompting AI with "tell me what you're uncertain about in this response" or "flag anything you're not sure of" often surfaces genuine uncertainty. But the things it remains confident about are not guaranteed to be correct — confidence is a stylistic output, not an accuracy signal. High-confidence answers still require verification for consequential claims.
5
Use grounded AI tools for research tasks.
Standard AI generates answers from memory. Retrieval-Augmented Generation (RAG) tools fetch information from specified sources before generating a response. This does not eliminate hallucinations but significantly reduces them for factual tasks — especially when the source material is known and trusted. See tools below.
6
Never use AI-generated content as sole evidence in a professional, legal, or medical context.
The Mata v. Avianca case is the floor, not the ceiling, of what AI hallucinations can cost in professional contexts. If AI output is going into a legal brief, medical recommendation, financial report, or published article, it requires the same verification standard as any other source — and typically a higher one, because it has been shown to hallucinate at measurable rates.
Tools That Reduce (Not Eliminate) Hallucination Risk
These tools add grounding mechanisms — connecting the AI to verified sources before generating answers. They significantly reduce hallucination rates for factual queries but do not eliminate them. All outputs still require judgment.
📓
Google NotebookLM
Upload your own documents; AI answers questions only against those sources. Dramatically reduces hallucination for document-grounded research. Shows citations by default.
FREE · notebooklm.google.com
🔍
Perplexity AI
Retrieves live web sources before answering and shows inline citations. Significantly better for current events and recent data than base models without web access.
FREE / $20 mo Pro · perplexity.ai
🤖
Claude with web search enabled
Anthropic's Claude with search grounding retrieves current information before responding. More likely to cite uncertainty when unsure than base GPT-class models.
FREE / $20 mo · claude.ai
⚖️
Harvey / CoCounsel / Lexis+ AI
Legal-specific AI tools that retrieve from verified legal databases (Westlaw, LexisNexis) before generating responses. Required standard for professional legal AI use.
ENTERPRISE PRICING · domain-specific
🧬
PubMed + AI wrapper
Tools like Elicit and Consensus retrieve from PubMed's indexed literature before generating responses. Significantly more reliable for medical and scientific citations than base models.
FREE / freemium · elicit.com / consensus.app
📊
ChatGPT with Code Interpreter + your data
When you upload your own data files and ask ChatGPT to analyze them, it works from the data you provided rather than its training memory. Much lower hallucination risk for quantitative analysis on provided datasets.
FREE / $20 mo · chat.openai.com
⚠
Even grounded tools hallucinate. RAG and web-search-enabled AI reduce hallucination rates significantly on factual queries, but they do not eliminate the problem. Retrieval can fail, sources can be wrong, and the model can still misread or misquote what it retrieved. Citation presence is a helpful signal, not a guarantee. Always verify citations grounded AI provides against the original source.
Will Hallucinations Be Fixed in Future Models?
Model improvements consistently reduce hallucination rates on benchmark tasks. GPT-4 hallucinates measurably less than GPT-3. Claude 3 hallucinates less than Claude 2. The trend line is positive. But several structural reasons suggest hallucinations will remain a feature of the technology for the foreseeable future, not a bug to be patched away:
First, the underlying architecture — next-token prediction — does not contain a truth mechanism. As long as models generate text by predicting statistically likely continuations rather than retrieving verified facts, plausible-but-wrong completions will always be possible. Second, the benchmark tasks used to measure hallucination cover well-documented domains; performance on obscure, niche, or recent topics does not improve at the same rate. Third, as models become more capable and their outputs more fluent, hallucinations may become harder to detect even as they become less frequent — a smaller rate on a more convincing foundation.
The EU AI Act (2024) specifically identifies hallucinations as a risk category requiring disclosure in high-stakes AI applications — an acknowledgment that regulatory bodies expect hallucinations to persist and require management, not elimination.
→
Deepfakes are a related trust problem — AI generating convincing false content in visual and audio form rather than text. The detection strategies overlap: verify the source, not just the content. See How to Spot a Deepfake for the visual and audio equivalent of this verification workflow.
The Essential Rule
AI that sounds certain is not AI that is correct. There is no correlation between how confident the output sounds and how accurate it is.
The standard of care for AI-generated factual content is: assume it may be wrong, verify what matters, and never let the fluency of the prose substitute for independent confirmation of the claims inside it. This is not a counsel of fear — it is a practical workflow for getting the productivity benefits of AI without the professional, legal, or personal consequences of treating its output as inherently reliable.
The tools getting better at reducing hallucination rates are genuine progress. But "lower hallucination rate" is not the same as "safe to trust without checking." Until AI architecture has a genuine truth mechanism — something fundamentally different from next-token prediction — verification remains the only reliable protection against hallucinations in any context where being wrong has consequences.
Sources & References
[1]
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., ... & Fung, P. — "Survey of Hallucination in Natural Language Generation" (ACM Computing Surveys, 2023). Comprehensive survey documenting hallucination rates across NLG tasks; ~19% rate in abstractive summarization studies. dl.acm.org/doi/10.1145/3571730 ↗
[2]
Alkaissi, H. & McFarlane, S.I. — "Artificial Hallucinations in ChatGPT: Implications for Scientific Writing" (Cureus, February 2023). Found 69% of AI-generated medical references contained significant errors. All citations appeared superficially plausible. ncbi.nlm.nih.gov/pmc/articles/PMC10008375 ↗
[3]
Mata v. Avianca, Inc., No. 22-cv-1461 (S.D.N.Y. June 22, 2023). Judge P. Kevin Castel's sanctions order against attorneys who submitted ChatGPT-generated fake case citations in federal court. $5,000 fine imposed. courtlistener.com ↗
[4]
Reuters — "Google shares dive after AI chatbot Bard flubs answer in ad" (February 8, 2023). Documents ~8% Alphabet stock decline (~$100B market cap loss) following hallucination in Bard's promotional demo. reuters.com ↗
[5]
Moffatt v. Air Canada, British Columbia Civil Resolution Tribunal, 2024 BCCRT 149 (February 14, 2024). Tribunal ruling holding Air Canada liable for its chatbot's incorrect bereavement fare policy information; "separate legal entity" defence rejected. canlii.org ↗
[6]
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., ... & Liu, T. — "A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions" (arXiv:2311.05232, November 2023). Comprehensive taxonomy of hallucination types and causes. arxiv.org/abs/2311.05232 ↗
[7]
Mallen, A., Shi, W., Michael, J., D'Oosterlinck, K., Alikhani, M., Hajishirzi, H., & Clark, P. — "When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories" (ACL 2023). Documents hallucination rates rising sharply for less-popular / obscure entities. aclanthology.org ↗
[8]
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Kiela, D. — "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (NeurIPS 2020). Foundational RAG paper demonstrating how source retrieval significantly reduces hallucination on knowledge-intensive tasks. arxiv.org/abs/2005.11401 ↗
[9]
OpenAI — "GPT-4 Technical Report" (2023). TruthfulQA benchmark results showing measured improvements over GPT-3.5 but persistent hallucination rates; discusses RLHF's role in reducing but not eliminating hallucination. arxiv.org/abs/2303.08774 ↗
[10]
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. — "On Faithfulness and Factuality in Abstractive Summarization" (ACL 2020). Documents ~30% faithfulness errors in neural summarization systems — early foundational evidence of structural hallucination in NLG. aclanthology.org ↗
[11]
European Parliament — Regulation (EU) 2024/1689 (EU AI Act). Article 13 (transparency) and Annex IV risk provisions identify hallucination as a disclosure requirement for high-risk AI systems in healthcare, legal, and educational contexts. eur-lex.europa.eu ↗
S
Sheldon Valentine
Founder · Dear Tech
Sheldon covers AI safety, practical risk assessment, and what the research actually says about how AI systems behave in the real world — not what the marketing says.
