A chatbot that gives a wrong answer does not set off an alarm. There is no error page, no failed payment, no angry email in most cases. The customer reads the answer, believes it or doubts it, and quietly makes a decision that you never get to see.
That silence is the real problem. This post explains how wrong answers turn into lost customers, why the losses are so hard to spot, and what you can do this week to find them.
Why chatbot mistakes are invisible
Most business systems fail loudly. A broken checkout throws errors. A down server triggers monitoring. A chatbot that confidently says the wrong thing looks, on every dashboard, exactly like a chatbot doing its job.
The conversation happened. The response time was fast. The customer did not ask for a human. By the usual metrics, that chat was a success.
What actually happened might be:
- The customer was told you do not ship to their country (you do), so they bought elsewhere.
- The customer was told a feature is "coming soon" (it already exists), so they picked a competitor.
- The customer was promised a discount that does not exist, then got angry at checkout.
- The customer asked a simple question, got a vague non-answer, and closed the tab.
None of these show up as failures unless someone reads the transcript and knows the right answer.
What a wrong answer looks like in practice
Here is a typical example from a business that sells online courses:
Customer: Do I get lifetime access if I buy the course? Chatbot: Access is valid for 12 months from the date of purchase.
The business actually offers lifetime access. It changed the policy a year ago and the old terms are still sitting in a forgotten FAQ page that the bot was trained on.
The customer was comparing two courses. Lifetime access was a deciding factor. They bought the other one. Nobody at the business will ever know that this sale was lost, because there is no record of a sale that never happened.
The four ways wrong answers cost money
Wrong answers do damage in a few predictable ways:
- Lost sales. Buyers who were ready to purchase get a reason not to. Wrong prices, wrong availability, wrong features and wrong policies all push people away at the exact moment they were close to deciding.
- Extra support load. Customers who get a wrong answer often contact support anyway to double-check, or come back later with a problem the bot created. The bot was meant to reduce tickets; now it is generating them.
- Honored mistakes. Sometimes you end up honoring what the bot said, either to keep the customer or because you have to. In Moffatt v. Air Canada (2024), a tribunal in British Columbia held the airline responsible for incorrect bereavement-fare information from its website chatbot and ordered it to compensate the customer. The argument that the chatbot was responsible for its own statements was rejected.
- Lost trust. Once a customer catches the bot in one mistake, they stop trusting anything it says, and often anything your website says.
A simple illustrative calculation
The numbers below are made up to show the shape of the problem. Plug in your own.
Say your chatbot handles 2,000 conversations a month. Say 20% of those are from people who are considering a purchase, so 400 buying conversations. If 5% of those get a wrong or unhelpful answer that ends the conversation, that is 20 lost opportunities a month.
If your average order is worth $150 and one in four of those people would have bought, that is 5 lost sales, or $750 a month. Over a year, around $9,000, from a tool you thought was working fine. And that excludes extra tickets, refunds and the customers who tell friends.
The point is not the exact figure. The point is that a small error rate multiplied by real volume becomes real money, and you cannot see it in your analytics.
How to find wrong answers before customers do
You do not need a big project to start. Here is a checklist you can work through today:
- List your top 20 customer questions. Pull them from support tickets, sales calls and the questions your team answers most.
- Write down the correct answer for each, with a link to the page or document that proves it.
- Ask your chatbot each question, in a few different phrasings. Customers do not use your internal vocabulary.
- Compare answers against the source, not against what "sounds right". A confident tone is not evidence.
- Check old content. Look for outdated pages, PDFs and FAQs the bot might still be reading. See outdated answers: when your chatbot lives in last year.
- Test the edges: refunds, cancellations, shipping exceptions, pricing tiers and anything with "it depends".
- Repeat after every change to your website, products or bot configuration.
Doing this by hand once is useful. Doing it every month by hand is where most teams give up.
Fixing what you find
When you find a wrong answer, fix the cause, not just the symptom:
- If the source content is wrong or outdated, correct or remove it.
- If the right information is missing, add a clear, specific page or knowledge base entry.
- If the bot is making things up when it does not know, tighten its instructions so it says what it does not know and offers a human or a link.
- Re-test the same question afterward to confirm the fix worked and did not break something else.
Find the leaks automatically
ChatLens360 reads your website and documents, generates realistic customer questions, asks your chatbot, and checks every answer against your own facts. You get a report showing each wrong answer with the full conversation, the expected answer, the source page and a ready-to-paste fix. Run a free chatbot audit to see what your bot is telling customers right now.