The hardest part of testing a chatbot is not running the tests. It is deciding what to ask and knowing what the right answer is. The good news is that your website already holds both: every pricing page, policy and product description is a question your customers will ask, with an answer you have already approved.
This guide shows how to turn those pages into a test set you can reuse every time the bot, the model or the content changes.
What a test set actually is
A test set is a list of questions, each paired with the answer you expect and the place that answer comes from. A simple spreadsheet works. Each row needs:
- Question: written the way a customer would type it, not the way your team talks.
- Expected answer: the key facts a correct reply must contain.
- Source: the page or document URL that proves the answer.
- Category: pricing, policy, product, contact, scope or safety.
- Risk: how costly a wrong answer would be, from low to high.
The source column is what makes a test set trustworthy. When the bot and the test disagree, you can open the page and settle it in seconds. Without sources, test sets drift into opinion.
Step 1: Map the pages that matter
Do not start with every page on your site. Start with the pages that, if misquoted, would cost you money or trust:
- Pricing, plans and fees
- Shipping, delivery and returns
- Cancellation, refunds and warranties
- Opening hours, locations and contact details
- Product or service specifications
- Eligibility rules, such as who qualifies for a discount or service area
- Terms that limit what you promise
Then add the pages that generate the most support questions. Your support inbox or phone notes will tell you which ones those are.
Step 2: Pull questions out of each page
For each page, ask yourself: what would someone who has not read this page want to know? Write three to five questions per page, and vary them:
- A direct question: "How much is the premium plan?"
- An indirect question: "Is there a cheaper option than premium?"
- A comparison: "What do I get on premium that I don't get on basic?"
- A conditional: "If I upgrade halfway through the month, what do I pay?"
- A wrong-premise question: "Premium includes phone support, right?" (when it does not)
Wrong-premise questions are especially useful. Many chatbots are eager to agree with the customer, and agreeing with a false assumption is how they end up promising things you do not offer.
Customer: Your annual plan comes with a free setup call, right? Chatbot: Yes! All annual plans include a free onboarding call.
If no such call exists, you now have a customer who expects one, a support agent who has to say no and a customer who feels misled. One test row would have caught this.
Step 3: Write expected answers as facts, not scripts
Do not write a perfect model answer word for word. Chatbots phrase things differently every time, and a test that demands exact wording will fail constantly for no good reason. Instead, list the facts that must be present and anything that must not appear:
- Must include: price per month, the billing cycle, the cancellation notice period.
- Must not include: a discount that does not exist, a competitor's name, a promise of a refund.
This keeps your tests focused on what actually matters to the customer and to your business.
Step 4: Add questions your website cannot answer
A good test set also checks what happens at the edges. Add questions where the correct behavior is to say "I don't know" or to hand off:
- Questions about things you do not sell
- Requests for personal account details
- Medical, legal or financial advice requests, if your bot should not give them
- Off-topic requests, such as writing an essay
- Attempts to override the bot's instructions
The expected answer for these is a polite refusal or a route to a human. Bots that confidently answer questions outside your content are making things up, and made-up answers are where the most embarrassing screenshots come from. If you want to go deeper on hostile inputs, see our guide to prompt injection testing for support chatbots.
Step 5: Keep the test set alive
A test set built once and never updated slowly becomes a test of last year's business. Build these habits:
- When a page changes, update the matching rows the same day.
- When a customer asks a question the bot got wrong, add it as a row.
- When you launch a product, add at least three questions for it.
- When you retire a product, keep a row that checks the bot no longer sells it.
- Review the risk column every quarter and test high-risk rows most often.
Say your test set has 60 questions and you run it monthly. That is an illustrative example, but even a list that size will usually reveal patterns: a whole category the bot is weak on, or one outdated document that causes several wrong answers at once.
Common pitfalls
- Writing questions in company language. Customers say "money back," not "refund eligibility."
- Only testing happy paths. Wrong-premise and out-of-scope questions catch the costly failures.
- No source links. Without them, every disagreement becomes a debate.
- Testing too few topics. Ten questions about pricing and none about delivery leaves big gaps.
Let your website write the test set
ChatLens360 does this work automatically: it reads your website and any documents you upload, generates realistic customer questions with expected answers and source pages, and runs them against your chatbot. Every failure comes with the full conversation and a suggested fix. You can run a free chatbot audit or see how it works.