Hallucinations test generative AI’s claim to accuracy

Pressure is building on developers and corporate users of generative AI as “hallucinations” — confident statements that are wrong — continue to surface in customer service chats, legal filings, coding assistants and financial research notes, exposing organisations to reputational harm, regulatory scrutiny and operational risk. The problem has become harder to dismiss as adoption accelerates, because the errors often arrive wrapped in fluent prose, invented citations and plausible-sounding numbers that can pass a quick skim.

Industry researchers and regulators increasingly treat hallucinations as a predictable failure mode rather than an occasional glitch. OpenAI has argued that standard training and evaluation practices can reward models for guessing instead of clearly signalling uncertainty, which helps explain why systems can sound assured even when the underlying answer is shaky. In parallel, the European Union’s AI Act is pushing providers and deployers towards tighter documentation, transparency and risk controls for powerful general-purpose models, adding compliance pressure to the business case for stronger reliability measures.

Hallucinations arise when a model produces output that is not grounded in verifiable data. In practical deployments, this can mean fabricating a court judgment, misstating a drug interaction, inventing a policy clause, or presenting an incorrect earnings figure with apparent precision. The risk is amplified in fast-moving domains where users expect up-to-date answers, yet many systems do not have reliable access to live information unless they are explicitly connected to vetted databases or retrieval tools. Even when connected, models can misread context, over-generalise, or “fill in” missing pieces.

Developers have responded by making factuality a more visible performance target. In several system cards and evaluation notes, OpenAI has described measuring hallucinations using question sets designed to elicit false claims, alongside accuracy scores that track whether answers are correct when the system attempts them. That shift reflects a broader industry trend: reliability is being treated as an engineering discipline with metrics, test suites and failure analysis, rather than a matter of user judgement alone.

Governments and standards bodies are also shaping expectations. A NIST profile accompanying the AI Risk Management Framework for generative systems describes methods to identify, measure and manage risks tied to content authenticity and other trust issues across the AI lifecycle, reinforcing the idea that hallucinations should be anticipated and controlled, not merely patched after high-profile mistakes. In Europe, the AI Act’s obligations and guidance for powerful models with systemic risks emphasise evaluations, adversarial testing, incident reporting and cybersecurity, signalling that regulators expect structured governance around model behaviour and downstream harms.

For businesses, the most common response has been to narrow the space in which a model is allowed to improvise. Retrieval-augmented generation — often called RAG — is now widely used to ground responses in company knowledge bases, product manuals, policy documents and approved research repositories. Instead of asking a model to “know” everything, RAG approaches prompt it to quote or summarise from retrieved passages and, in well-designed systems, to decline when relevant documents cannot be found. Guardrails are being paired with retrieval: instruction hierarchies, refusal rules, banned-claim lists and content filters that block risky outputs before they reach a customer or employee.

Another layer involves verification. Teams are increasingly inserting automated “checks” between a model’s draft answer and the final response: cross-referencing claims against structured databases, requiring citations to internal documents, or running a second model to flag unsupported statements. In regulated workflows, a human-in-the-loop review remains the default for high-impact decisions, with AI used for drafting, triage and summarisation rather than final determinations. That approach can reduce risk, but it also adds cost and latency — a trade-off many firms now accept for sensitive use cases such as healthcare, employment and financial advice.

Model makers, for their part, are exploring ways to align a system’s confidence with its actual likelihood of being correct. One theme in research is improving how systems communicate uncertainty, so that “I don’t know” becomes a rewarded outcome when evidence is weak, rather than a failure. Another is better domain-specific training and evaluation, because hallucination behaviour can vary sharply between general knowledge questions and specialised fields like chemistry, law or accounting, where small errors can carry outsized consequences.