Skip to content
ChatLens360
Chatbot testing

Regression testing for AI chatbots: why last month's good answers break

Chatbots that answered correctly last month can quietly start getting things wrong. Regression testing catches the slide before customers do.

Illustration of chat bubbles being checked off with green ticks for the article "Regression testing for AI chatbots: why last month's good answers break", tagged Regression testing, Test automation, Chatbot accuracy.

You tested your chatbot at launch, fixed the obvious mistakes and moved on. Then, months later, a customer forwards a screenshot of the bot quoting a price you changed long ago, or refusing to answer a question it used to handle well. Nothing in your setup seemed to change. So what happened?

This is a regression: something that used to work has stopped working. With AI chatbots, regressions are common, quiet and often caused by changes nobody connected to the bot.

Why chatbot answers change without anyone touching the bot

Traditional software only changes when someone ships new code. AI chatbots have more moving parts, and many of them move on their own schedule:

  • Model updates. Providers update or retire models. A new version can phrase things differently, follow instructions more or less strictly, or handle edge cases in new ways.
  • Content changes. Someone edits the returns page, removes a FAQ or uploads a new PDF. The bot's knowledge shifts, sometimes in ways that conflict with older documents it still has.
  • Prompt edits. A teammate tweaks the system prompt to fix one problem and accidentally weakens another rule.
  • Retrieval changes. If your bot searches a knowledge base, changes to how documents are split or ranked can change which facts it sees.
  • Integration changes. A tool the bot calls, such as an order lookup, changes its format or starts timing out.

Each of these can turn a correct answer into a wrong one, and none of them sends you an alert.

What a regression looks like in practice

Customer: Do you ship to Canada? Chatbot: Sorry, we only ship within the United States.

Last month the same bot correctly said you ship to Canada with a flat fee. Since then, someone uploaded an old shipping PDF from before you expanded. Now the bot has two conflicting sources and picks the wrong one. Every Canadian visitor who asks is told to shop somewhere else. You will not see this in your sales numbers as an obvious drop; you will just see fewer orders from a region and wonder why.

That is the real cost of regressions: they rarely look like a crash. They look like slightly worse results that are easy to blame on something else.

How regression testing works

Regression testing is simple in principle. You keep a fixed set of questions with known good answers, and you run them again on a schedule or after every change. Then you compare:

  1. Baseline. Run your test set and record which answers pass. This is your known-good state.
  2. Re-run. After a change, or on a fixed schedule, run the exact same questions again.
  3. Compare. Look for tests that passed before and fail now. Those are your regressions.
  4. Investigate. For each regression, find the cause: new content, a prompt edit, a model change.
  5. Fix and confirm. Correct the cause, re-run and check the test passes again.

The comparison is the important part. A single test run tells you how good the bot is today. Comparing runs tells you whether it is getting better or worse, and exactly which answers moved.

What to include in a regression suite

Your regression suite does not need to cover everything. It needs to cover the answers you cannot afford to get wrong:

  • Prices, fees and plan details
  • Refund, return and cancellation rules
  • Delivery areas and timeframes
  • Safety rules, such as refusing to give medical or legal advice
  • Instructions the bot must keep, like not sharing internal information
  • Questions that failed before and were fixed, so they do not quietly break again

That last group is often forgotten. Every bug you fix should become a permanent test. Otherwise the same mistake can return with the next content update and you will fix it twice.

When to run regression tests

There are two kinds of triggers, and you want both:

  • Event-based: after editing the system prompt, changing the AI model, uploading or removing documents, or redesigning key website pages.
  • Scheduled: weekly or monthly, to catch changes you did not make yourself, such as provider-side model updates.

If you are about to switch models, our guide on testing a chatbot after a model upgrade covers what to check first.

A simple regression routine to start today

  • Pick 20 to 50 questions that cover your highest-risk answers.
  • Write the facts each correct answer must include.
  • Run them once and save the results with today's date.
  • Add a calendar reminder to re-run them monthly and after any bot change.
  • Keep a short log: date, what changed, which tests flipped from pass to fail.
  • Turn every customer-reported mistake into a new test.

Even done by hand, this routine catches regressions far sooner than waiting for complaints.

Catch regressions automatically

ChatLens360 runs scheduled audits on your chatbot, compares each run with the previous one and emails you when answers that used to pass start failing. Each regression comes with the conversation, the expected answer and the source page, so you can see what changed. You can run a free chatbot audit to set your baseline.

Put these ideas into practice

Run an automated audit of your own chatbot for free.