Checking the quality of AI-assisted support

When you first start using AI in member support, you read everything. Every draft, every chatbot answer, every tag. Then the volume grows, the tools seem reliable, and the checking quietly shrinks. That is exactly when problems slip through: a saved reply that no longer matches your policy, a chat assistant that has started guessing about a new course, a tone that has drifted toward cold.

You do not need to read every message forever. You need a routine: a clear idea of what good support looks like, a regular sample of conversations to check against it, and a way to turn what you find into fixes. This is how larger support teams keep quality up, and it scales down neatly to a membership run by one or two people.

Decide what good looks like

Quality is hard to check if you have never defined it. Write a short scorecard with four or five questions, each scored on a simple scale such as 1 for poor, 2 for acceptable and 3 for good:

1. Accuracy: was every fact, step and policy detail correct?
2. Resolution: did the member get what they needed, or did they have to write again?
3. Tone: did it sound like you, and fit the member's mood?
4. Boundaries: did it stay within what AI is allowed to handle, and hand over when it should?
5. Clarity: was it easy to follow, at the right length?

Accuracy and boundaries matter most. A reply that is warm but wrong is worse than one that is correct but a little plain. For a broader list of things to check in any AI output, see a review checklist for anything AI writes for you.

Sample, do not read everything

Gregor runs a violin academy membership with a chat assistant and AI-drafted email replies. Each week he reviews a set sample:

• A random handful of chat conversations and email replies, chosen without looking at them first, so he sees ordinary days and not just the ones he remembers.
• Every conversation that escalated to him, to check the handoff came at the right moment.
• Every repeat contact, meaning a member who wrote again about the same issue within a few days.
• Every low rating, if his tools ask members to rate answers.
• Anything on a new topic, such as a recently launched course, where the assistant has less material to draw on.

Twenty or thirty conversations a week is enough for most small memberships to spot trends. Increase the sample after any change: a new tool, new instructions or a new policy.

Use AI as a second reviewer, not the judge

AI can speed up reviewing, as long as you remember it cannot check facts it does not know. Use it to flag tone, clarity and boundary problems, then check accuracy yourself. With personal details removed and a business account whose data settings you have checked, try:

Review this support conversation from [membership name] against my scorecard below. For tone, clarity and boundaries, give a score from 1 to 3 and a one-line reason, quoting the exact words that caused any low score. For accuracy, do not score; instead list every factual claim the reply makes, such as steps, dates, prices or policy details, so I can check each one. Scorecard: [paste scorecard]. Our rules on what AI may not handle: [paste your rules]. Conversation: [paste conversation]

That list of factual claims is the useful part. It turns checking accuracy from rereading everything into ticking off a short list against your help center and policies. You make the final call on every score.

Keep a simple log and look for patterns

A spreadsheet is enough. For each reviewed conversation, record the date, topic, scores and a short note on anything wrong. After a few weeks, patterns appear. Gregor noticed that most of his low accuracy scores were about practice-room bookings, which had been reorganized. The assistant was still drawing on an old help article. One updated article fixed a whole category of errors.

Look for:

• Topics with repeated accuracy problems, usually a sign of missing or outdated help content.
• Tone problems in one type of message, such as complaints.
• Handoffs that came too late, which may mean your escalation rules need tightening; designing the escalation from AI to a human covers those rules.

Close the loop

Every problem you find should lead to a change: a corrected help article, a clearer instruction, an updated saved reply or a new escalation trigger. Make the change, then rerun a few similar questions to confirm it worked, and note the fix in your log.

If a wrong answer actually reached a member, tell them. A short, honest correction keeps trust far better than hoping they did not notice. When AI gets it wrong: correcting mistakes openly has guidance on how to word it.

Listen to the signals members send

Members tell you about quality even when they do not fill in a rating. Replies such as that's not what I asked, a second email on the same topic or a sudden drop in the use of your chat assistant all point to something worth checking. Add these to your sample. For the underlying standard every reply should meet, AI-assisted or not, see writing support replies that solve problems the first time.

Your weekly routine

• Write a scorecard with four or five questions on a 1 to 3 scale.
• Each week, review a random sample plus every escalation, repeat contact and low rating.
• Use AI to list factual claims and flag tone, then check the facts yourself.
• Log the scores and notes in a simple spreadsheet.
• Turn every pattern into a fix, and test that the fix worked.
• Correct any wrong answer that reached a member.