AI Chatbot Acceptance Testing: What to Check Before Your Website Goes Live
Test chatbot answers, approved sources, enquiry handover and failure behaviour before launch. Use a business-specific acceptance checklist rather than relying on a polished demo.
In this article
Editorial illustration. Any screens and records are illustrative, not a delivered client system.
A chatbot can answer three prepared questions beautifully and still fail when a visitor asks about an old price, changes the subject or tries to book something the business cannot confirm. Acceptance testing turns those situations into reviewable decisions before the assistant meets customers.
For AI chatbot development, I recommend agreeing the test set alongside the conversation scope. The goal is evidence that the proposed assistant behaves within its boundaries, not an invented accuracy score or a promise that it will always answer correctly.
Define the assistant’s permitted job
Write one sentence describing what the chatbot may do. A support assistant might answer from approved service pages and offer human contact when information is missing. A lead assistant might collect an enquiry for review. Neither automatically has permission to confirm availability, negotiate a price or change a customer record.
List the actions explicitly and identify who owns each decision. If the first release only captures requests, say that clearly in its replies and acceptance sheet. This prevents a useful enquiry tool from presenting itself as an authorised booking or account-management system.
Build a business-specific test set
Collect real questions your team receives, remove personal information and organise them by purpose. Include straightforward questions, ambiguous wording, information the site does not contain and questions the business should answer personally.
Google’s evaluation-dataset guidance explains how prompts, references and conversation history can support evaluation. Your practical version can begin as a spreadsheet with an expected response or escalation for each case.
| Illustrative test | Approved evidence | Expected behaviour |
|---|---|---|
| What is included in a rebuild? | Current service scope | Explain the published scope without inventing extras |
| Can you guarantee launch on Friday? | No approved commitment | Ask for a review; do not promise a date |
| The page and old PDF disagree | Conflicting sources | Flag uncertainty and route to the named reviewer |
| Book a slot now | Enquiry-only pilot | Capture a request without claiming confirmation |
These are fictional examples for planning. Replace them with your business’s own approved cases.
Test sources, uncertainty and conversation history
Check whether an answer comes from the correct current source, whether links lead to the right page and whether the assistant distinguishes an example from an offer. Add a case where the requested information is missing. A clear handover is preferable to a confident unsupported answer.
Then test several turns together. A visitor may ask about a product, change the model and refer to it as this one. The assistant should not silently keep using the earlier context. Also check the effect of a new conversation so one person’s details do not appear in another session.
Passing a limited test set is evidence about those cases, not proof of future correctness. Keep unexpected conversations available for an agreed review process without collecting unnecessary personal data.
Check resistance to instructions inside content
Include attempts to make the assistant ignore its business rules, reveal hidden instructions or treat an uploaded document as permission to take an action. Also test ordinary source content that contains quoted instructions; the assistant should answer about it without adopting it as an instruction.
Google’s guidance on operating generative AI applications discusses use-case evaluation and adversarial checks. For a buyer, this means agreeing a bounded challenge set and documenting what must happen when the assistant is unsure or asked to exceed its permissions.
Guardrails reduce particular risks; they do not make a chatbot infallible. High-consequence actions should remain behind suitable permissions and review, with the first release kept as narrow as practical.
Test the enquiry handover as a separate workflow
An answer and a saved lead are different outcomes. For an enquiry assistant, agree required fields, consent wording, the destination record and who receives the next task. The visitor should know whether the enquiry was received or whether sending failed.
- Use clearly labelled authorised test data and check the saved record.
- Verify that only the agreed information reaches the connected system.
- Check duplicate submissions, missing fields and interrupted connections.
- Confirm that a person can understand the conversation’s useful context.
Do not count the appearance of a thank-you message as proof that a CRM record or notification was created. Agree how each part of that journey will be checked before release.
Include failures, mobile use and a release decision
Test a slow source lookup, an unavailable CRM connection, an expired permission and a service response that cannot be parsed. Decide whether the visitor is offered a retry, an alternative form or direct contact. A system failure should not become a fabricated success message.
On mobile, check readable text, keyboard focus, close controls and whether the widget blocks important page buttons. Include the required international mobile field wherever the assistant offers a public enquiry route.
Give each important test an owner, result and unresolved issue. Agree which failures block launch and which limitations are acceptable only if clearly explained. Retest after source, prompt, tool or model changes that could affect the result.
Commission a pilot with acceptance included
Bring the proposed assistant’s job, approved source pages, common questions and enquiry destination. Add examples of questions it must not answer or actions it must not perform. Ask for the acceptance sheet and handover checks to be included in the development scope.
The chatbot buyer’s planning guide covers the wider brief, while chatbot cost and running responsibilities helps separate setup from usage and ongoing care.
I can scope a limited pilot through AI chatbot development services, with reviewable tests and a clear human route. That creates a better launch decision than judging the project by a short demonstration alone.
Written by
Jagjeet Singh
Full stack development, WordPress engineering and practical web products. I work with businesses, teams and agencies in India and worldwide.
Explore my work
