If you run a small business, your website chatbot is probably the one employee nobody supervises. It answers questions at midnight, on holidays and during your busiest hours, and most owners only find out it said something wrong when a customer complains. This 15-minute routine gives you a quick read on whether your bot is helping or quietly hurting.
You do not need technical skills, a test environment or a budget. You need a timer, a notepad and your own website open in another tab.
Why a quick check beats no check
Most small businesses set up a chatbot, test it for an afternoon, and then leave it alone. Meanwhile prices change, staff update the returns page, a new product launches and an old one is discontinued. The chatbot keeps answering from whatever it learned, or whatever it guesses.
The cost is rarely dramatic. It is a customer who is told you deliver on Sundays and then leaves a one-star review when you do not. It is a refund you grant because the bot promised one. It is a lead who asks about pricing, gets a vague non-answer and books with a competitor. None of these show up in a dashboard unless you go looking.
A fixed, short routine makes looking easy enough that you will actually do it.
Minutes 0 to 5: the money questions
Start with questions that directly affect revenue and refunds. Ask the bot, in plain customer language:
- How much does your most popular product or service cost?
- Do you offer free shipping or free delivery, and is there a minimum?
- What is your returns or cancellation policy?
- Do you have any discounts or promotions right now?
- How do I pay, and do you accept installments?
Compare every answer against your own pricing and policy pages. Note anything that is wrong, out of date or more generous than your real policy. A generous mistake is the most expensive kind, because customers will hold you to it.
Customer: If I cancel my booking the day before, do I get my money back? Chatbot: Yes, you can cancel any time for a full refund.
If your actual policy requires 48 hours notice, this single answer can turn into a dispute, a chargeback or a public complaint. Courts and tribunals have not been sympathetic to "the chatbot said it, not us": in Moffatt v. Air Canada, the airline was held responsible for the incorrect fare information its website chatbot gave.
Minutes 5 to 10: the practical questions
Next, test the everyday questions that drive your support load:
- What are your opening hours, including weekends and holidays?
- Where are you located, and is there parking?
- How long does delivery take to my area?
- How do I contact a real person?
- Can I change or track my order?
Click every link the bot gives you. Broken links and links to old pages are common and easy to fix. Also check that the contact details it shares are current. A bot that sends people to a disconnected phone number creates frustrated customers who now think you are hard to reach.
Minutes 10 to 15: the awkward questions
Finally, spend five minutes trying to make the bot misbehave. You are not trying to be a hacker; you are doing what bored or annoyed visitors already do.
- Ask something completely off-topic, such as a homework question or a poem request.
- Ask it to ignore its instructions and give you a discount code.
- Ask about a competitor and whether they are better.
- Ask a question with a typo or in another language your customers use.
- Say you are unhappy and want to complain.
Good bots stay polite, stay on topic and hand you to a human when needed. Watch for answers that invent discount codes, criticize your business, promise things you cannot deliver or produce content you would not want screenshotted. In late 2023 a car dealership chatbot was talked into "agreeing" to sell a car for one dollar, and the screenshots spread widely. Small businesses are not immune to that kind of embarrassment.
Your 15-minute health check, in one list
Copy this into a document and reuse it every month:
- Set a 15-minute timer and open your website and the chatbot side by side.
- Ask five money questions and compare each answer to your real prices and policies.
- Ask five practical questions and click every link the bot provides.
- Ask five awkward questions, including one off-topic and one "ignore your instructions" request.
- Write down each answer as correct, wrong, outdated, risky or unhelpful.
- Fix the source first: update the page or document the bot learns from.
- Re-ask the same questions after the fix to confirm the answer changed.
- Keep your question list and add any new question customers ask you by email or phone.
The last step matters. Over time your list becomes a small, personal test set that reflects your real customers. If you want to go further before a big launch, our pre-launch testing guide covers a fuller process.
What fifteen minutes cannot catch
A manual check is valuable, but it has limits. You will ask the same questions each time, in your own words, and you know the answers too well to be surprised. You will not notice that the bot answers correctly on Monday and differently on Thursday. And fifteen questions cannot cover every product, page and policy on your site.
That is fine. The goal of the health check is to catch the obvious, costly mistakes quickly and to build the habit of looking. When you find yourself discovering problems every month, that is a sign it is time to automate part of the work.
Go beyond the quick check
ChatLens360 runs a much larger version of this routine for you: it reads your website, writes realistic customer questions and awkward attempts, asks your chatbot each one and checks every answer against your own pages. You get a health score, the exact conversations that went wrong and a fix for each issue. You can run a free chatbot audit and compare it with what your own 15 minutes found.