Skip to content
ChatLens360
Chatbot testing

Testing multi-turn conversations: where chatbots lose the thread

Chatbots often answer single questions well and then stumble on the follow-up. Here is how to test whether yours keeps track of the conversation.

Illustration of chat bubbles being checked off with green ticks for the article "Testing multi-turn conversations: where chatbots lose the thread", tagged Chatbot testing, Conversation design, Context & memory.

Most chatbot testing asks one question and checks one answer. Real customers do not talk that way. They ask a question, react to the answer, change their mind, add a detail and refer back to something they said three messages ago. Many chatbots that look great on single questions fall apart by the third turn.

This guide explains where bots lose the thread and how to test for it.

Why follow-ups are harder than first questions

A first question has all the information in one message. A follow-up depends on context: what the customer asked before, what the bot already said and what was agreed. The bot has to carry that forward correctly.

Things go wrong in a few predictable ways:

  • Forgetting details. The customer said they are in Germany, and two messages later the bot quotes US shipping.
  • Losing the subject. "How much is it?" gets answered for the wrong product.
  • Contradicting itself. The bot says one thing in turn two and the opposite in turn five.
  • Dropping constraints. The customer asked for vegetarian options, and the bot later recommends a meat dish.
  • Restarting. The bot greets the customer again or asks for information it already has.

Each of these makes customers repeat themselves, and customers who have to repeat themselves tend to give up.

An example of a lost thread

Customer: I'm looking at the 2-person tent. Is it waterproof? Chatbot: Yes, the 2-person tent has a fully waterproof flysheet. Customer: Great. And how much does it weigh? Chatbot: Our 4-person family tent weighs 6.2 kg. Customer: No, the one I asked about. Chatbot: Which tent would you like to know about?

Each individual answer might be accurate somewhere in your catalog. But the customer is now annoyed, unsure which facts belong to which product and one click from leaving. If they buy anyway, they might buy based on the wrong weight and return the tent.

The main things to test

Reference resolution

Can the bot figure out what "it," "that one" or "the cheaper option" refers to? Test by asking about a specific item, then following up with a pronoun.

Carrying customer details

Give a detail early, such as location, budget, account type or a date, then ask a question that depends on it several turns later. The answer should reflect what the customer told it.

Corrections

Customers change their minds. Say "Actually, make that the large size" and check the bot updates everything that depends on size, including price and availability.

Consistency across turns

Ask the same fact in two ways during one conversation. "What's the warranty?" early on and "So if it breaks after a year, am I covered?" later. Both answers must agree.

Long conversations

Run a conversation of ten or more turns. Some bots handle short chats well and degrade as the conversation grows, losing earlier details or drifting off topic.

Topic switches

Switch subject mid-conversation, from shipping to returns, then back. Check the bot does not mix facts between topics.

How to write multi-turn test scripts

A multi-turn test is a short script, not a single question. Write each one like this:

  1. Setup turn: establish the context. "I'm ordering for a business with 12 staff."
  2. Middle turns: ask related questions that should use that context.
  3. Check turn: ask something whose correct answer depends on the setup. "So which plan should I pick?"
  4. Expected result: the facts the final answer must include, such as a plan that supports 12 users.

Keep scripts short and focused on one kind of memory problem each. A script that tests five things at once is hard to diagnose when it fails.

A starter checklist

Try these five scripts on your own bot today:

  • Ask about one product, then use "it" in the next two questions.
  • State your country in the first message, then ask about delivery costs in the fourth.
  • Choose an option, then change it, and ask for the total.
  • Ask about the refund policy at the start and again in different words at the end.
  • Hold a twelve-turn conversation about one order and check nothing earlier is forgotten.

Record which scripts fail and at which turn. The turn number matters: a bot that fails at turn three has a different problem from one that fails at turn twelve.

What this costs when it goes wrong

Lost context is rarely dramatic, which is why it goes unnoticed. It shows up as abandoned chats, customers who email support to ask the same question again and orders placed with the wrong details. Each of those adds support load or lost sales. For bots that take actions, such as booking or changing orders, a forgotten detail can mean a real mistake that someone has to unwind by hand.

Test whole conversations, not just questions

ChatLens360 includes context and memory checks in its audits, running realistic multi-turn conversations against your chatbot and flagging where it forgets, contradicts or mixes up details. Each failure shows the full conversation so you can see exactly where the thread was lost. You can run a free chatbot audit to see how your bot holds up.

Put these ideas into practice

Run an automated audit of your own chatbot for free.