AI hallucination: 17% to 33% wrong in a Stanford test

AI hallucination has public measurements, and the most solid number I know comes from a test in which the AI had every reason to get it right. Researchers at Stanford’s RegLab and Institute for Human-Centered AI (HAI) ran 202 legal questions, registered before the test, through three legal research tools that search a database of statutes and case law before writing. Each one gave incorrect information or cited a source that did not support its claim in 17% to 33% of answers, and a general-purpose model with no access to the database reached 43% (Magesh et al., 2025). These numbers hold for that test, run in 2024, with those versions and that kind of question, because there is no single error rate for generative AI. The measurements show that the rate depends on the task, on whether the AI has the source at hand and on whether someone checks the answer, and that is what sales and service leaders need to bring into the operation.

At Iasbeck&Co we implement CRM and AI agents in B2B operations, and the question I hear most from executives after the first demo is how often AI gets it wrong. I gathered the most cited public measurements and translate the numbers into proposals, pricing and commercial policy.

What is AI hallucination?

One of the most cited literature reviews on the subject, published in ACM Computing Surveys, defines hallucination as generated content that is nonsensical or unfaithful to the provided source and separates two types. Intrinsic hallucination contradicts the source, like a summary that changes the year of a fact from the original text. Extrinsic hallucination adds something the source can neither confirm nor refute, and the authors note that this information may even be correct, but it calls for caution because nobody can verify it against the source material (Ji et al., 2023).

NIST, the US government’s technical standards body, uses the term “confabulation” in its generative AI risk profile and describes it as erroneous or false content stated with confidence. The document links the phenomenon to the design of the models, which generate text by predicting the next word, and says the risk grows in long answers and in specialized domains (NIST, 2024). For anyone who sells, the consequence is that AI writes a nonexistent discount with the same fluency it uses for the right price.

Which studies measure how often AI is wrong?

I chose measurements that describe their method. One of them is a leaderboard run by a company that sells its own evaluation model, and the AI Index compiles third-party data.

Source	Year	Sample and method	Main finding
Magesh et al., Journal of Empirical Legal Studies	2025 (tests in 2024)	202 preregistered legal questions, run through three tools that search their own databases and through a general-purpose model; manual evaluation	Legal tools hallucinated in 17% to 33% of answers; the general-purpose model, in 43%
Vectara, Hallucination Leaderboard	updated 10/16/2026	Each model summarizes more than 7,700 articles using only the text; a company model judges faithfulness	1.8% to 24.2% across 109 models
Stanford HAI, AI Index Report 2026, ch. 3	2026	Compiles third-party tests and a survey of business leaders run with McKinsey	Without a source, 26 models hallucinate 22% to 94% of the time; 74% of organizations see inaccuracy as a relevant risk
Mata v. Avianca, US District Court for the Southern District of New York	2023	Opinion and order on sanctions in case 22-cv-1461	Six nonexistent opinions cited in a filing; US$ 5,000 penalty
Ji et al., ACM Computing Surveys	2023	Literature review on hallucination in text generation	Defines intrinsic and extrinsic hallucination
NIST AI 600-1	2024	Generative AI risk management profile	Classifies confabulation as a risk specific to generative AI and links it to automation bias

How often is AI wrong in the Stanford test?

The Stanford study tests the setup companies are buying now: a language model that queries a document database before answering. The providers advertised that the method eliminated or avoided hallucination, and the authors registered questions and protocol on March 22, 2024, before any test. Of the 202 questions, 80 are general legal research, such as US bar exam questions, 70 depend on jurisdiction or recent changes in law, 22 carry a built-in false premise and 30 cover simple facts about cases (Magesh et al., 2025).

The study’s criterion works for any field. An answer counts as hallucinated when it contains incorrect information or when it claims that a source supports something the source does not support. The second case is the most treacherous, because the cited document exists. The paper names the products; here I number them, because what matters to leadership is the pattern.

System tested	Correct and well grounded	Incomplete or refused	With hallucination
Legal tool 1	65%	18%	17%
Legal tool 2	42%	25%	33%
Legal tool 3	20%	63%	17%
General-purpose model, no legal database	49%	8%	43%

Source: Magesh et al., 2025, figure 4.

What the table teaches anyone buying AI

Searching the database reduced hallucination compared with the general-purpose model, and no tool came close to zero. Tool 3 hallucinated little because it refused or left incomplete 63% of its answers, since it only queries articles written by the provider’s own team, and a low error rate can hide a tool that hardly ever answers. Tool 2 wrote the longest answers, averaging 350 words against 219 and 175 for the others, and the authors note that long text carries more claims that can be wrong and takes longer to check. They point out the limits: three products, a small sample and a snapshot in time, since the answers of one tool changed during the research, carried out between March and May 2024.

What does the Vectara hallucination leaderboard measure?

The Vectara leaderboard is the most frequently updated public measurement. The company gives each model more than 7,700 articles from news, science, medicine, law, business and other fields, from 50 to 24,000 words long, and asks for a summary using only the information in the text. A commercial evaluation model from Vectara itself, HHEM-2.3, decides whether each summary is faithful to the original. In the October 16, 2026 version, the 109 models listed range from 1.8% to 24.2% hallucination, and by my count the median of the table is 9.6% (Vectara, 2026).

The design has limits the company itself acknowledges. Summarizing a text that sits in front of the model is the most favorable situation possible, and one model judges another, with no human review of each case. The open version of the evaluator works only in English, the set of texts is not public and the company changed that set between versions of the leaderboard. The AI Index reproduces the leaderboard, with rates from 1.8% to 5.4% among the top 15 (Stanford HAI, 2026). I use the number as an order of magnitude for summarization with a source, without extending it to other tasks.

Why is AI wrong more often when it has no source?

The same AI Index chapter brings a measurement with the opposite design. AA-Omniscience, from the company Artificial Analysis, asks 6,000 knowledge questions in six fields, from law and health to software engineering, with no supporting text, and its scoring penalizes wrong answers without penalizing refusals, to encourage the model to admit it does not know. Across 26 models, hallucination rates ranged from 22% to 94%. The report also highlights KaBLE, an academic test with 13,000 questions: recent models score 95% when a false belief is attributed to another person, and 62.6% when the users themselves state it (Stanford HAI, 2026). The salesperson who asks how to justify the 30% discount the policy allows, when the policy stops at 15%, hands the AI a false premise it tends to accept.

Measurement	Task	Does the AI get the source?	Hallucination range
Vectara, 2026	Summarize an article	Yes, the full text	1.8% to 24.2%
Magesh et al., 2025	Answer a legal question	Yes, through database search	17% to 33% with database; 43% without
AA-Omniscience, in the AI Index 2026	Answer a knowledge question	No	22% to 94%

The three scales use different criteria and cannot be compared point by point, as the AI Index itself warns. The order, however, repeats: with the whole text in front of it, the error stays below 10% for a little over half of the models; with a search that can bring back the wrong document, it rises; with no source, the answer depends on what the model memorized.

What did the court decide in Mata v. Avianca?

The case that took hallucination beyond technical circles is a passenger’s lawsuit against an airline, heard in New York. On March 1, 2023, the plaintiff’s lawyers filed an opposition to the request to dismiss the case as time-barred, citing court decisions. The defense said it could not find several of those cases, and the lawyers submitted texts of “opinions” generated by a general-purpose AI chatbot. On June 22, 2023, Judge P. Kevin Castel identified six nonexistent opinions, attributed to real judges, imposed a US$ 5,000 penalty on the two lawyers and the firm and ordered them to send the decision to the client and to each judge falsely named as the author of a fake opinion (Mata v. Avianca, 2023).

The reason for the sanction is what matters to leadership. The judge writes that there is nothing improper about using a reliable AI tool for assistance and that the rules give lawyers a gatekeeping role over the accuracy of what they sign. The penalty came because nobody checked the citations and because the lawyers stood by the fake opinions after the court questioned their existence. One of them testified that he was working under the false perception that the tool could not fabricate cases. NIST calls this behavior automation bias, the tendency to over-rely on an automated system, and warns that it amplifies the risk of confabulation (NIST, 2024).

How concerned are companies about AI inaccuracy?

In a survey by the AI Index and McKinsey of business leaders across regions and industries, with no respondents from China, the share that considers AI inaccuracy a relevant risk rose from 60% in 2024 to 74% in 2025. Inaccuracy became the most cited risk among those the report shows, ahead of cybersecurity (72%) and regulatory compliance (63%), and the share that says it actively mitigates this risk went from 55% to 71% (Stanford HAI, 2026). These are perception numbers, with no sample size reported in the chapter. Other risk data from the global surveys are in state of AI in companies.

Where can AI be wrong in sales and customer service?

In the operations I work with, AI makes mistakes where a new salesperson would, only with confidence and at volume. In Ji et al.’s terms, an intrinsic error is the call summary that says the customer approved the budget when they asked for more time, and an extrinsic error is the proposal that adds an implementation deadline found in no source at all.

Where AI writes	Typical error	Cost of the error	Control
Sales proposal	Price, discount, scope or deadline off the price list	Lost margin or contractual commitment	Values come from CRM fields
Commercial and service policy	Outdated refund, SLA or cancellation terms	Complaint and legal risk	Answer only with the passage from the current policy
Meeting summary	Objection or next step the customer never stated	Wrong sales forecast	Transcript passage for each claim
Product specification	Nonexistent feature or integration	Sale that turns into churn	Product base with an owner and review date
Account research	Made-up job title, revenue or news	Outreach based on wrong data	Required source for each external data point

In the proposal, the control that cuts the most errors is taking the job of writing numbers away from the AI: price, discount and scope come from the CRM, and the AI handles the surrounding text. The step-by-step for a proposal is in B2B sales proposal.

What is RAG, in plain language?

RAG stands for retrieval-augmented generation, the technique used by the legal tools tested at Stanford. In class, I compare it to an open-book exam: instead of answering from memory, the system first searches the company’s base for passages that seem to answer the question, hands those passages to the model and asks it to write from them.

The Stanford study shows where the open-book exam still fails: the search brings back a similar text that does not apply to the case or a repealed version of the rule, or the model reads the right passage and draws the wrong conclusion from it (Magesh et al., 2025). In a company, RAG shifts much of the problem to the base, like the old discount policy nobody archived. The prerequisites of the base for customer service are in AI in customer service.

Why human review needs a method

Checking takes time, because each claim and each citation needs verification. In an illustrative example, an agent that answers a thousand questions a month with a 5% error rate produces 50 wrong answers in the same confident tone as the right ones. That is why I ask for short answers, with the source passage next to each claim, and the instruction to say it found nothing when the base does not have the answer. What the reviewer checks before sending should be written in the AI use policy.

What these numbers do not say

None of these measurements gives you the AI error rate in your company. Stanford used US legal questions, in English, with 2024 versions; Vectara measures only summarization with the text at hand, judged by a model; AA-Omniscience measures knowledge without a source, with its own scoring rule.

The unit also changes. Stanford counts answers with at least one hallucination, so a long answer with one wrong sentence counts as a whole, while other tests count claims. Two measurements come from companies, Vectara and Artificial Analysis, which calls for attention to the method without discarding the data. The Mata v. Avianca case illustrates how the error happens without measuring how often. And the numbers age fast, because each leaderboard update brings new models that may be wrong more often than the previous ones. In the searches I did for this text, I found no primary source for any general rate of the “AI hallucinates in X% of answers” kind; what exists are rates per task.

What leadership does with these numbers

This is the plan I propose before letting AI write anything that reaches the customer.

1. Map where AI writes for the outside. List proposals, follow-ups, ticket replies and any text with price, deadline or terms, and rank each one by the cost of an error.
2. Measure your operation’s rate. Build 50 to 100 real questions, with the right answer written before the test, as Stanford did, and run the agent on them before going live and after every change of model or base.
3. Provide the source and forbid answers without one. Connect the agent to the price list, the current policy and the product base.
4. Take the numbers out of the AI’s hands. Price, discount, deadline and scope come from CRM fields.
5. Review according to the cost of the error. Price, terms and policy go through a person before sending; internal summaries are checked by weekly sampling, counting wrong claims.
6. Give the base and the agent an owner. One person answers for the knowledge base and another for the agent, and both receive the monthly error rate, as in the design of AI agents.

What HR and L&D need to consider

The lawyer’s testimony in Mata v. Avianca shows the training gap: he knew how to use the tool and did not know it could invent an opinion with a docket number and a judge’s name. What reduces the automation bias described by NIST is the practice of verification, in which the team learns to ask for the source, to check whether the cited passage says what the answer claims and to distrust premises built into the question itself. HR measures the effect by the rate of errors that reached the customer, before and after the training.

Public measurements show that AI with a source is wrong less often than AI working from memory and that none has reached zero, and the result depends on the base the company maintains and on the method of whoever checks. That is the work of the in-company applied AI training, built on the company’s own processes and data, with groups for executives, leaders and sales, marketing and CS teams. It delivers the prompt library by function and the agent architecture, with what each agent reads, what it does on its own and who approves the rest, including the rules for sources and review.

Frequently asked questions

What is AI hallucination?

It is content generated by AI that is wrong or not supported by the source, presented with the same confidence as a correct answer. The literature separates hallucination that contradicts the source from hallucination that adds something impossible to verify, and NIST uses the term confabulation for the same phenomenon.

How often is generative AI wrong?

It depends on the task and on whether the AI has the source. In the text summarization measured by Vectara, rates range from 1.8% to 24.2% across 109 models; in legal questions with database search, Stanford measured 17% to 33%; in open questions with no source, the AI Index reports 22% to 94%. Your operation’s rate only shows up in a test with your own questions.

Does RAG eliminate AI hallucination?

No. In the Stanford test, tools that searched a legal database were wrong less often than a general-purpose model and still hallucinated in 17% to 33% of answers. RAG reduces errors when the base is up to date and the search brings back the right document, which is why the quality of the base becomes the main control point.

Why does AI make things up so confidently?

Models generate text by predicting the next word from the patterns they learned, and that prediction can produce a fluent and false sentence. The confident tone is the same in both cases, and NIST warns that this leads people to believe the content and act on it.

Is the company liable for the AI’s errors?

In Mata v. Avianca, the court sanctioned the lawyers and the firm because it was their job to check what they signed. Beyond that case, liability depends on the contract, the industry and each country’s law, and the company’s legal team should take part in defining the review rules.

How do I prepare my team to review what AI writes?

Train verification along with use: asking for the source of each claim, checking whether the cited passage supports the sentence, distrusting premises built into the question and knowing what never goes out without review, such as price, deadline and policy. Measure, before and after, the rate of errors caught in review and the rate of errors that reached the customer.