If you missed Part 8, start there. That article covered the collaboration loop between Support, Product, and Engineering. This part moves closer to the floor, where managers coach agents and QA teams decide what good performance looks like when AI sits inside the daily workflow.
AI changes what managers need to coach
Most QA programs were built for a simpler world.
An agent read the ticket, wrote the answer, used the macro, followed policy, and closed the case. Managers reviewed the result.
AI changes that sequence.
Now the agent may be reviewing a draft, correcting a summary, checking a source, judging risk, and deciding whether the answer is safe enough to send. That means QA cannot only score the final reply anymore.
It has to score the decision behind the reply.
A manager used to review a ticket and ask familiar questions:
Did the agent answer the customer?
Was the tone appropriate?
Was the macro used correctly?
Was the ticket solved within SLA?
Those questions still matter. They are no longer enough.
When agents work with AI every day, performance includes a new layer of judgment. The agent is no longer writing every sentence from scratch. They are reviewing, verifying, correcting, escalating, and deciding when AI output is safe enough to use.
That changes the coaching model.
A fast reply that repeats a wrong AI answer is not good performance. A polished response that skips source verification is not good performance. A clean escalation with no evidence is not good performance. A perfect tone on top of a bad decision is still a bad ticket, despite humanity’s brave ongoing effort to confuse style with substance.
In the AI era, QA has to measure judgment, not just output.
The old QA rubric misses too much
Traditional QA rubrics usually score things like accuracy, tone, policy adherence, documentation, and resolution. Those categories still have value, but they were built for a world where the agent was the primary producer of the answer.
AI changes the work in five ways:
The agent must verify AI output before using it.
The agent must know when not to trust a confident answer.
The agent must connect customer language to internal systems and policy.
The agent must document enough evidence for other teams to act.
The agent must escalate when the risk exceeds their authority.
If the rubric does not score those behaviors, managers end up coaching the wrong thing.
The team gets faster, but risk goes up. Customers receive confident answers with weak grounding. Escalations become cleaner on the surface, but less useful underneath. QA celebrates tone while the operating system quietly leaks trust from every seam, like a product demo held together by hope and quarterly planning.
The new standard: agents as AI reviewers
The support role is shifting from answer writer to answer operator. That does not mean agents stop writing. It means writing is only one part of the job.
The agent now has to ask:
Is this answer grounded in the right source?
Does the customer’s situation match the policy conditions?
Did AI miss a key constraint?
Is this safe to send as written?
Does this need escalation, or is it within my authority?
Will the next person understand what happened?
That is the work managers need to coach.
Good AI-era support performance means the agent uses AI to move faster without transferring risk to the customer.
That standard matters because AI does not remove accountability. It changes where accountability shows up. The answer may start with AI, but the decision still belongs to the support team.
A practical QA rubric for AI-era support
Here is a simple rubric managers can use to coach agents in an AI-enabled support environment.
1 means the behavior was missing.
2 means the behavior was inconsistent or incomplete.
3 means the behavior met the standard.
4 means the behavior exceeded the standard and created reusable value.
The difference between coaching and catching
Bad QA says, “You missed this.”
Good coaching says, “Here is the decision point you missed, and here is how to spot it next time.”
That distinction matters. If managers only correct outcomes, agents learn to fear QA. If managers coach decision points, agents learn how to think.
AI-era coaching should focus on the moment before the mistake.
For example, the issue is not only that the agent sent the wrong answer. The issue is that they trusted an AI summary without checking the source article. Or they saw a policy-heavy question and treated it like a simple how-to. Or they escalated a ticket without explaining the business impact.
The coaching target is the decision pattern.
Instead of saying:
“This answer was wrong.”
Say:
“The answer became risky at the moment the customer mentioned an exception. That was the signal to verify against the policy source before replying.”
Instead of saying:
“You should have escalated this.”
Say:
“This crossed from routine support into eligibility interpretation. When a customer’s outcome depends on policy conditions, escalation is part of the resolution path, not a failure.”
Instead of saying:
“This escalation was unclear.”
Say:
“The internal team needed the account ID, timeline, system behavior, and the exact decision needed. Without those, they have to investigate from scratch.”
This is how QA becomes skill development, not just a scorecard. Revolutionary stuff, apparently. Teach the behavior before everyone spends six months admiring the dashboard.
What managers should stop rewarding
Speed alone is a dangerous metric in AI-enabled support.
AI makes speed easier. That is useful when the work is low-risk and the source is clear. It is dangerous when the case requires judgment.
Managers should stop rewarding:
Fast replies with weak verification.
High solve counts with high reopen rates.
Clean tone with poor policy accuracy.
Escalations that move work without moving clarity.
AI usage volume without evidence of better outcomes.
The goal is not slower support. The goal is better control.
Fast replies with weak verification are still bad support. AI-era QA has to score the decision behind the reply.
Speed is valuable only when the answer is verified, safe, and useful.
This is where many teams get uncomfortable. They bought AI to move faster, then act surprised when speed without judgment creates more cleanup work. An entire industry keeps rediscovering that “faster wrong” is not a strategy.
What managers should reward instead
Managers should reward the behaviors that make AI safer to use in production. That means looking past the final reply and asking what the agent did before they sent it.
Verified answers
Reward agents who show their work. The best agents do not blindly accept AI output. They compare it against the system of record, the policy, the knowledge base, and the customer’s actual situation.
Clean judgment calls
Reward agents who know when a case is routine, when it is risky, and when it needs escalation. This is especially important for policy, money, identity, access, eligibility, and legal-adjacent language.
Useful escalations
Reward agents who make internal teams faster. A strong escalation saves time because it includes the issue, evidence, impact, owner, and requested decision.
Customer clarity
Reward agents who explain complex situations in plain language. The customer should not need a support background, a legal degree, and an emotional support spreadsheet to understand the answer.
Feedback that improves the system
Reward agents who flag broken macros, bad AI answers, missing knowledge, confusing routing, and repeated failure patterns. In AI-enabled support, the best agents do not only close tickets. They improve the system that closes tickets.
A simple coaching model for managers
Managers need a repeatable way to coach AI-era performance. Here is a practical model.
Step 1: Identify the decision point
Find the moment where the agent had to choose. Did they trust AI, verify a source, escalate, use the right tool, explain uncertainty, or protect sensitive data? Do not start with the final answer. Start with the decision that shaped the answer.
Did they trust AI?
Did they verify a source?
Did they escalate?
Did they use the right tool?
Did they explain uncertainty?
Did they protect sensitive data?
Do not start with the final answer. Start with the decision that shaped the answer.
Step 2: Name the risk
Every coaching conversation should identify the risk category.
Common categories include:
Wrong source
Missing condition
Unsafe promise
Weak escalation
Sensitive data exposure
Misread customer intent
AI hallucination
Policy edge case
Poor handoff
The point is not to create a taxonomy museum. The point is to make patterns visible.
Step 3: Show the better move
Give the agent the replacement behavior.
For example:
“When the answer depends on eligibility, verify the system status and policy source before using AI language.”
Or:
“When AI gives a confident policy answer, check the date, source, and exception conditions before sending.”
Or:
“When escalating, include impact, evidence, account details, and the exact decision needed.”
Agents need concrete moves, not motivational fog.
Step 4: Connect it to the customer outcome
Show why the behavior matters.
The customer avoids repeat contact. The internal team receives a cleaner escalation. The company avoids a wrong promise. The AI loop gets better evidence. The next agent inherits a better ticket.
That connection turns QA from criticism into operational learning.
A lightweight QA scorecard
Here is a compact scorecard managers can adapt.
Verification
Standard:
The agent checks AI output against the appropriate source of truth before replying or escalating.
Evidence:
Ticket notes, source links, system checks, policy references, runbook references.
Miss signal:
AI language appears in the customer reply, but the ticket shows no source verification.
Judgment
Standard:
The agent identifies risk level and chooses the correct handling path.
Evidence:
Risk tier, escalation reason, decision note, verified customer context.
Miss signal:
A high-risk case receives a routine answer or generic macro.
Communication
Standard:
The customer receives a clear, accurate, plain-language response with the right next step.
Evidence:
Reply addresses the actual issue, explains constraints, avoids unsupported promises, and sets expectations.
Miss signal:
The reply sounds polished but does not answer the customer’s real question.
Escalation
Standard:
The escalation includes enough context for the next owner to act without rediscovery.
Evidence:
Problem statement, customer impact, account details, timeline, source checks, requested action.
Miss signal:
The escalation says “please review” and then everyone gets to enjoy a fresh round of corporate archaeology.
Safe tool use
Standard:
The agent uses AI and internal tools within approved boundaries and protects sensitive information.
Evidence:
No restricted data entered into unsafe tools, redactions completed, AI used only for approved tasks.
Miss signal:
The agent uses AI for a restricted decision or exposes sensitive data in the wrong place.
Calibration matters more now
If managers do not calibrate, QA becomes unfair.
One reviewer may score AI-assisted replies harshly because they distrust the tool. Another may ignore AI risk because the final answer sounds clean. A third may reward speed because the ticket closed quickly.
That creates chaos.
Calibration gives managers a shared standard. It also protects agents from inconsistent coaching.
A simple calibration process works:
Review five tickets each week across different risk levels.
Score them independently using the rubric.
Compare where reviewers disagree.
Identify the decision point behind each score.
Update the rubric language when confusion repeats.
Share one coaching takeaway with the team.
The goal is not perfect scoring. The goal is consistent judgment.
If two managers score the same ticket differently, the answer is not to debate vibes until morale collapses. The answer is to clarify the standard.
Make AI performance visible without making agents paranoid
Agents need to know AI-related performance will be reviewed. They also need to know the review is fair.
That means leaders should be explicit about what QA is measuring.
Tell agents:
We are not scoring you for using AI.
We are scoring how you verify, adapt, and apply AI output.
We are not expecting perfection.
We are expecting safe decision-making.
We will coach patterns, not punish isolated confusion.
We will use QA findings to improve training, knowledge, prompts, and workflows.
This matters because agents will hide mistakes if QA feels punitive. Hidden mistakes break the AI feedback loop. Then leaders sit in meetings wondering why no one flagged issues earlier, because apparently fear remains a poor operating system.
What good looks like
Here is a simple example.
A customer asks whether they qualify for a program benefit after a policy change. AI drafts a confident answer based on the knowledge base. The agent notices the article references last year’s rule and the customer’s account shows a different status. The agent checks the policy source, confirms the eligibility condition, rewrites the response, and adds an internal note explaining the source mismatch. They also flag the stale article for review.
Under the old QA model, the score might focus on whether the final reply was accurate and polite.
Under the AI-era model, the score includes more:
The agent caught stale AI-supported guidance.
The agent verified the source of truth.
The agent matched the rule to the customer’s account.
The agent avoided an unsupported promise.
The agent created a feedback signal to improve the system.
That is what good looks like.
Not because the agent used AI perfectly. Because the agent reviewed AI responsibly.
The manager’s weekly coaching rhythm
Managers do not need a massive QA overhaul to start. They need a weekly rhythm.
Monday: Pick the focus
Choose one AI-era skill for the week.
Examples:
Verifying policy answers
Writing better escalation notes
Spotting hallucinated AI language
Protecting sensitive data
Handling uncertainty
Confirming customer intent before using AI drafts
Keep the focus narrow. A team cannot improve everything at once, despite leadership’s touching belief in magic.
Tuesday to Thursday: Review live examples
Pull a few real tickets. Look for the decision point, not only the final response.
Ask:
Where did the agent rely on AI?
Where did verification happen?
Where did risk increase?
Where did the customer need clearer language?
Where did escalation help or hurt?
Use real examples with names removed when sharing with the team.
Friday: Coach one pattern
Share one pattern with the team.
Example:
“This week we saw AI give strong answers on policy questions, but agents sometimes missed effective dates. Starting next week, any policy-based AI draft must be checked against the effective date before sending.”
Then update the rubric, macro, article, or training note if needed.
Coaching without system improvement is just commentary with a calendar invite.
The leadership mistake to avoid
The biggest mistake is treating QA as an agent-only process.
If QA keeps finding the same AI-related miss, the problem may not be the agent.
It may be:
The knowledge base is stale.
The prompt is too broad.
The macro lacks conditions.
The workflow routes too late.
The escalation path is unclear.
The policy source is hard to find.
The AI tool makes unsupported claims too easily.
A strong QA program does not only ask, “Did the agent perform?”
It also asks, “What did the system make easy, and what did the system make risky?”
That question separates real AI operations from theater.
The operating principle
AI-era QA should make performance visible, teachable, and fair.
Visible means managers see the actual behaviors behind the outcome.
Teachable means agents understand the decision pattern and the replacement behavior.
Fair means the team gets a clear standard, consistent coaching, and a system that improves when patterns repeat.
The goal is not to turn agents into auditors. The goal is to help them become reliable operators of an AI-enabled support system.
That is the next support skill.
Not typing faster.
Not trusting AI more.
Not distrusting AI by default.
The skill is knowing what to verify, when to slow down, when to escalate, and how to turn messy customer work into clean operational signal.
A practical starting point
Start with five QA questions.
Did the agent verify AI output against the right source?
Did the agent identify the risk level of the request?
Did the agent communicate the answer clearly and safely?
Did the agent escalate with enough evidence for action?
Did the agent use AI and internal tools within guardrails?
Then take ten recent tickets and score only those five things.
You will learn fast where the real coaching gaps are. You will also learn where your system is making good agents look worse than they are.
That is useful. Annoying, yes. But useful.
Closing
Part 8 focused on the collaboration loop across Support, Product, and Engineering. Part 9 brings that loop into daily management.
Because the loop only works if agents know what to flag, managers know what to coach, and QA knows what good looks like.
The old question was:
“Did the agent solve the ticket?”
The better question now is:
“Did the agent make a sound decision with AI in the workflow?”
That is the standard worth coaching.
If you had to add one behavior to an AI-era QA rubric, what would you score first: verification, judgment, communication, escalation, or safe tool use?




