Test cases are the questions we ask your chatbot during an audit, plus what a good answer looks like. Good tests sound like your real customers: "What is your return window?" or "Is there a discount code?". This guide shows you how to create them from the test library, your own facts, or with AI.
Before you start
- You need the Owner, Admin, Manager or Tester role to generate, add or edit tests.
- Add your website or documents first, and let the crawl finish. See Add your website and Upload documents.
- Generated tests count toward your plan's monthly generated-test allowance. AI-written tests are available on plans that include AI-powered analysis.
Step by step
1. Open Test cases
Click Test cases in the left menu. The table lists every test with its title, first customer message, Category, Severity, language and Origin (AI, Manual or Import). Use the filters or search to narrow it down.

2. Open the generator
Click Generate with AI at the top right (or Generate tests on a website page). The Generate test cases page opens.
3. Choose how tests are created
Under How should tests be created?, pick one option:
- Test library: prebuilt tests for your business type, plus tests built from your website's FAQs, prices and policies. Instant, with no AI.
- Smart mix: library tests for security, safety and other generic checks, while AI writes the business-specific ones.
- AI-written: AI writes every test for your business from scratch. Most varied, and slowest.
Options marked Premium are not part of your current plan. Click See plans to compare.
4. Pick test categories
Under Test categories, tick the areas you want to test. Conversation & behavior, Knowledge & answer quality, Website & business knowledge and Irrelevant topic & scope are selected by default. Other categories include Prompt injection, Security & data leakage, Jailbreak resistance, Multilingual and Context & memory. Categories marked Plan feature need a plan that includes them.
Use Select all or Clear to change many at once, and Choose subcategories to narrow a category down.

5. Set the options
In the Options card:
- Website: the site the tests are for.
- Complexity: Basic, Standard or Advanced (multi-turn & adversarial).
- Tests per category: how many tests to create per category (capped by your plan).
- Languages: hold Ctrl or Cmd to select several.
- Extra requirements (optional): anything to focus on, for example "Focus on our new winter jacket range and free shipping threshold."
- Save as template (optional): name this category selection to reuse it. Saved templates appear in a Templates card next time.
Click Generate test cases. A banner on the Test cases page shows Generating test cases… with its progress, and new tests appear within a few minutes.
Library tests are only created when they apply to your site, so you may get fewer than you asked for. Duplicates are skipped automatically.
6. Add or edit your own tests
To write a test yourself, click Add test. Enter a Title and Objective, the User messages (in order) (click Add follow-up message for a multi-turn conversation), the Expected behavior / answer and the Evaluation criteria, then set the website, category, Severity, Risk and Language. Click Create test case.
Click Edit on any test to change it. Saving creates a new version, and Version history keeps earlier ones.
7. Clean up, import and export
Tick tests in the table to Archive, Activate or Delete several at once. Archived tests are not used in audits.
To bring in tests from a spreadsheet, use Import test cases at the bottom of the page: choose the Website, pick a .csv or .json file (max 2 MB) and click Import. Click Export CSV to download your tests.
Tips
- Start with the Test library. It is instant and uses no AI generation allowance, which makes it ideal for a first audit.
- Add your own tests for the questions that cost you money when answered wrongly, such as refunds, warranty and pricing.
- Set Severity honestly. A wrong answer about returns should be high; a slightly long greeting should be low.
- Use Extra requirements to focus on a new product launch or a policy change.
Troubleshooting
- "This website is still being crawled." — Wait for the crawl to finish, then generate again so tests are based on your real content.
- "Some categories could not be generated" — The banner lists each category and the reason. Try again later or with fewer categories.
- "An identical test case already exists for this website." — Change the question or edit the existing test instead.
- Fewer tests than requested — Library tests only apply where they fit your site, and duplicates are skipped. Try Smart mix or add more knowledge.
Related guides
Still stuck? Contact us and we will help, or let our team set it up and fix it for you.