Think back to school. An open-question exam, and you hit one you don't know. What do you do?
You write something. Of course you do. A blank line scores zero for certain, while half a page of confident prose about roughly the right subject might still catch a mark from a tired examiner. So you use the right vocabulary, you sound sure of yourself, and you carefully avoid saying anything specific enough to be marked wrong.
That is also a fairly precise description of a hallucinating chatbot, and it is not a coincidence. OpenAI's 2025 paper Why Language Models Hallucinate claims that models are trained and graded on tests where a wrong answer and an "I don't know" score are exactly the same. Under that scoring, guessing is the better strategy. So the model guesses. Not because it is broken because it learned what we taught it, which is that sounding certain pays.
The bar is a colleague, not a test-taker
When you hire someone, you are not looking for a good exam candidate. You want the colleague who says "honestly, I'm not sure, but Marijke in legal deals with this" or better, "that's in the 2024 policy doc, second half, let me find you the link."
That person is more useful than the one who invents an answer. And they are more useful precisely because you can trust the answers they do give.
Almost no chatbot is built to that bar, and it costs more than it looks. Most notoriously, Air Canada's chatbot invented a bereavement refund policy that did not exist. A customer relied on it, the airline refused to honour it, and a tribunal held Air Canada liable ruling that it made no difference whether the wrong information came from a web page or a bot. The airline had argued the chatbot was a separate legal entity responsible for its own actions. The tribunal was not impressed.
That is the version that makes the news, which makes it easy to file under someone else's bad luck. The common outcome is quieter. The bot is wrong often enough that people stop trusting it, so they check its answers against the source anyway, which is the work it was bought to remove. Tickets it was meant to deflect come back a week later, now starting with "your chatbot told me". Nobody files a bug, because nothing is broken exactly. It just gets used a little less each month until it is decorated.
First, the uncomfortable part about your data
Most chatbot quality problems are content problems wearing a costume.
If your knowledge base is a folder of PDFs last updated in 2023, three contradictory versions of the same policy, and a wiki nobody has touched since the author left, no amount of clever engineering saves you. The expert colleague is only as good as what they studied, give them last year's notes and they will confidently teach you last year's rules. This is the least fun finding and the most common one, and it is rarely fixable in the same quarter, because content cleanup is an organisational project involving people who do not care about your chatbot.
So the useful question is not "how do we fix the data first". It is: how do we build something that fails gracefully on imperfect data, and how do we find out how bad it currently is?
Start with the mechanical half, which is much easier. Ask what happens when someone drops a PDF into your system. The answer is often nothing or worse than nothing: the file is accepted, no text comes out of it, and the bot cheerfully answers questions about a document it has never seen a word of. Scanned paperwork is the classic case. A PDF exported from Word has a text layer you can read; a PDF that came off a scanner is a stack of photographs of paper. In a file browser they look identical, and they behave nothing alike.
So read everything, and say clearly what you could not read. Take the text layer wherever there is one PDF, Word, PowerPoint, Excel, HTML. Use a model to read the page images only where there is no text layer, and label that source when you do, because a model asked to read a page returns a faithful-looking rendering, which is exactly the thing you cannot check a quote against afterwards. And report per file rather than per batch: someone drags in eleven documents, one of them is a .doc from 2004, and the other ten must still land. None of that is interesting work. A surprising share of "the bot doesn't know about X" turns out to be X sitting in a file that silently produced nothing.
Teaching it to say "I don't know"
If you want a bot that admits uncertainty, something has to reward admitting it. That is the lesson of the exam, applied: score a wrong answer as worse than a blank one, and "I'm not sure, but here is the policy document" becomes the winning move rather than the losing one. OpenAI illustrated this exact dynamic in their 2025 paper on why models hallucinate. If an AI is asked for an obscure birthday, taking a wild guess offers a 1 in 365 chance of lucking into a point, whereas admitting "I don't know" guarantees a zero on standard accuracy scoreboards. To demonstrate the fix, OpenAI tested benchmark scoring like the penalty metrics in SimpleQA that dock points for confident errors while leaving abstentions neutral. The moment a wrong guess carries a negative score, the leaderboard flips: models programmed to acknowledge uncertainty instantly outperform those that gamble, turning restraint into the highest-scoring move on the test.
Then design the refusal properly, because this is where most attempts fall down. A bot that says only "I cannot help with that" is worse than useless users learn to rephrase until they trick it into answering, and now you have a bot that bluffs on demand. A bot that says "I don't have that in my sources, but it is covered in the employee handbook, and Support can confirm" is honest and genuinely useful in the same breath. Handing off well is a feature, not an admission of defeat.
Make it show the page
Most chatbots do a token version of this: a list of documents under the answer. That is a bibliography, not a citation; it tells you which books were on the desk, not which sentence was read. It is also easy to produce while still making the claim up, because that list comes from the search index rather than from the answer.
The version worth having works differently. Cut the material into numbered passages, and require every claim to carry two things: the passage it came from, and the exact words. Then this is the part that changes the product look those words up in the stored document after the answer is written.

There are only four possible outcomes, and all four are worth showing the reader. The words are in the source, character for character. The opening is, and the model shortened the rest. The passage is real but those words are not in it. Or the passage never existed at all. That last one is the entire point. Before you check, "according to the discovery notes" and a completely invented set of discovery notes look identical on screen. Afterwards they do not, and anyone can tell at a glance which sentences in an answer are load-bearing. The quote should also be the link. Click it and you land in the document at that spot, highlighted, with the text around it readable because the question you actually want answered is whether the quote still means that in context.

The half nobody demos
Everything above is about the answer. A depressing amount of whether a chatbot survives its first year is about everything around the answer.
An expert colleague can be corrected without a code deploy, and your chatbot should work the same way. If changing a prompt, adding a document or fixing a wrong answer needs an engineer and a release, the content will rot because the people who know the content are not the people with deploy access.
You also need to see what it is doing. Tokens, latency and cost per client, visible without asking an engineer, is what turns "it feels slow lately" into a number, and "is this still worth having on" into a decision.
And the two cheapest wins in how good it feels have nothing to do with correctness: stream the answer, so the first words arrive in a moment instead of the whole paragraph in four seconds, and offer two or three follow up questions, because most people have no idea what the bot can actually do.
Five questions worth asking
You do not need to be technical to find out where a chatbot stands. Ask it directly, or ask whoever is selling you one.
- When it doesn't know, what does it say? Ask something just outside its material. Fluent and confident is a failing grade.
- Where did that come from? Try to get from a claim to the sentence behind it. A filename does not count.
- What happened to the scanned documents? Read properly, skipped in silence, or read by a model and labelled as such?
- Who fixes a wrong answer, and how fast? If it takes an engineer and a release, the content will rot the people who know the content are not the people with deploy access.
- What is the number? If quality is not measured, every argument about it is a matter of taste, including the ones about budget.
That is an afternoon's work on a system you already own. The uncomfortable part is that the answers tend to be consistent: a bot that fails the first question usually fails the rest, because they are all the same discipline seen from different sides.
Change the exam
We ended up with confidently wrong chatbots because the tests we grade them on reward confident wrongness. If you want an expert instead of a practised bluffer, you have to build the assessment that rewards being one: correct when it knows, quiet when it doesn't, and useful about where to look next. It is the cheapest thing on the list and the only one that compounds.
We handle AI chatbot development and maintain AI features in production, usually for clients who need them to work with real data and real users. Are you already on this path, or stuck on the data challenge? Let’s look at it together.
.avif)




