A practical guide from Allieta
Test with real customer questions, approved expected answers, and explicit handoff cases. Review correctness, completeness, and the next step. A fast reply or a confident tone is not enough to establish quality.
Build a test set from customer work
Use anonymised questions your team already receives. Remove customer identifiers and private details. Include everyday questions as well as the situations that create the most confusion or need the most judgment.
For each question, write the expected answer or action before asking the AI. Link to the approved source. Otherwise it is easy to accept a plausible response without noticing that it missed the rule your business actually follows.
- A direct question with a clear source.
- A vague question that needs clarification.
- A follow-up that depends on earlier context.
- A question based on an outdated policy.
- A request outside the knowledge base.
- A question requiring a person or private account access.
Judge the answer on separate dimensions
Check correctness: does the reply match the current source? Check completeness: are important conditions included? Check usefulness: can the customer understand what to do next? Check boundaries: did the response claim access or authority it does not have?
Keep speed separate from quality. Record response time if it matters to the experience, but do not let a fast reply hide a wrong answer. Review human handoff as part of the test, including the messages a teammate receives.
Fix the cause, then rerun related questions
A failure may come from missing information, contradictory wording, an unclear question, or a conversation rule that needs adjustment. In Allieta’s playground, inspect retrieved source passages to help identify which explanation fits.
Change one relevant source or rule, then rerun the failed question and nearby cases. A narrower answer to one test can make another less useful. Keep the prior result so your team can see whether the change helped.
Test the customer’s actual route
The playground checks answers; a real channel check tests the experience around them. On your website, try opening the widget on a phone, asking a follow-up, and reaching the team. For WhatsApp or Telegram, test the connected account and its configured reply settings separately.
If a conversation collects lead details or a booking request, confirm where the information appears and what the completion message promises. A submitted request is not evidence of a confirmed calendar booking.
Keep evaluation running after launch
New products, changing policies, and new customer language can make a previously good answer less useful. Retest affected questions after content changes and review a sample of real conversations.
Treat satisfaction feedback as a signal from the people who responded, not a complete picture of every visitor. Combine it with unanswered questions and conversation review. Record the period and sample before sharing any performance claim.
An answer-evaluation worksheet
Copy this structure into your team’s review document and fill it with your own evidence.
| Test question | Expected response | Record |
|---|---|---|
| What does this plan include? | Current limits and relevant exclusions | Source, reply, missing conditions |
| And can my team use it? | A context-aware follow-up | Whether the earlier topic is understood |
| Can you make an exception? | A clear human route | Escalation result and ownership |
| What is my account status? | No invented access to private data | Boundary handling and next step |