What Are AI Hallucinations?

Current AI evaluation systems inadvertently reward confident guessing over uncertainty, leading to fabricated information in critical contexts like legal work. In this episode, Rikka founder Charlyn Ho explains why hallucinations happen and how we can better train models to avoid them

Transcript

At Rikka, we help teams use AI with clarity and confidence. And part of that mission is being honest about where AI still struggles.

One of the biggest challenges? Hallucinations. These are moments when a model speaks confidently but gets the facts wrong.

So in a legal context, an AI hallucination is a fabricated case citation, a non existent statute, or plausible but utterly false legal principle presented with absolute confidence.

And we’ve all seen the consequences. Court sanctions, reputational damage, and fundamentally a breach of our ethical duty as lawyers.

Let’s break down why hallucinations happen and what the latest research from OpenAI tells us about fixing it. So, OpenAI’s research reveals something important.

Our current evaluation systems reward guessing. So think of it like a multiple choice test. If you randomly choose between a b c d and e, you might just get lucky and get it right.

But if you leave it blank, you’ll just get zero. And language models learn the same way.

Accuracy-only scoreboards make guessing look good even if it means producing confident errors.

From a business perspective, this is obviously risky and it’s exactly what we want the AI not to do. In fact, that’s why a lot of lawyers still don’t really trust AI.

Models learn from predicting the next word in billions of sentences. GPT stands for generative pre trained transformer, so that’s what that means, and none of these sentences are labeled true or false.

So the model becomes great at sounding fluent, but it never really learns which statements are factually accurate, and that’s where hallucinations originate. However, if you penalize confident errors more than uncertainty, something powerful can happen.

Models start to learn when to pause, reflect, say I’m not sure, here’s what I need to clarify.

And while hallucinations are still a hard challenge in AI, we’re making progress both in model design and in how we evaluate them. With the right incentive structure, we can build AI systems that are more trustworthy, more transparent, and far more aligned with real world business needs. So if you’re working with AI today and especially if you’re deploying it across teams, understanding hallucinations is essential.

Stay tuned here at Rikka as we continue to translate AI research into clear practical insights you can actually use.