42% deflection. The honest number was 19%.

The COO opened the call with a good number and a bad feeling. Six months after launch the copilot was reportedly deflecting 42% of inbound support contacts, comfortably beating the 30% business case. Headcount had been held flat against 20% contact growth. On paper it was the most successful project in the company.

The bad feeling came from everywhere else. CSAT was down four points year on year. The support team had quietly started telling customers how to skip the bot. Two senior agents had resigned, and both exit interviews mentioned it.

When the people closest to a system don't believe its dashboard, the dashboard is usually the thing that's wrong. They see the failures; leadership sees the aggregate.

I asked for the metric definition. It took two days to find someone who could produce it, which is itself diagnostic. It read, in effect:

# as implemented
deflected = session_started_with_bot
            AND NOT agent_ticket_created_in_same_session

Read that carefully. A contact counts as deflected if the customer talked to the bot and did not reach an agent in that session. Which means every one of these was a success:

• The customer got a useful answer and left. Genuinely deflected.
• The customer gave up and closed the tab. Counted as deflected.
• The customer gave up and phoned instead. Counted as deflected.
• The customer gave up and came back angry the next morning. Counted as deflected, twice.
• The customer churned. Counted as deflected, permanently.

The metric could not distinguish a solved problem from an abandoned customer. Worse, it was monotonically improved by making the bot harder to escape, and the team, doing exactly what they were measured on, had spent six months making the handoff to a human progressively more buried.

I replaced it with a definition that requires evidence of an outcome rather than evidence of an absence:

# resolution, not containment
resolved = bot_handled_contact
           AND NOT repeat_contact_same_issue_within_72h
           AND NOT channel_switch_within_24h      # chat → phone → email
           AND NOT negative_csat_on_session

Three properties matter more than the exact thresholds:

• It is cross-channel and cross-session. A customer who gives up on chat and phones you is no longer invisible. This single change accounted for most of the gap.
• It requires a positive signal, not the absence of a negative one. Silence is not success.
• Making escape harder now makes the number worse, because trapped customers come back. The incentive points the right way.

Recomputed against six months of history, the real figure was 19%. That conversation was not fun. It was also the most valuable hour of the engagement, because every improvement after it was aimed at a real failure rather than a reported one.

With resolution measurable per issue type, the failures sorted themselves immediately.

Billing was the whole story. A fifth of contacts, resolving at six percent, and the cause was not model quality: the bot understood the disputes perfectly. It had read access to the billing system and no write access, so its best possible outcome was to explain the problem back to the customer and then fail to fix it. That is a more infuriating experience than no bot at all.

Write access, bounded

Credits under a threshold, issued by the bot directly, logged and reversible. Above it, escalation. The threshold was set from the existing distribution of agent-issued credits, an amount agents already approved without review 96% of the time. Nothing about the risk posture changed; the work simply stopped requiring a human to retype it.

Escalation the customer doesn't feel

The old handoff dropped the agent into a fresh window with the transcript on a different tab. Customers repeated themselves every time, which is the single most reliable way to make someone hate a support experience. The rebuilt handoff passes the agent a summary of what was tried, what was ruled out, and the customer's actual goal, and the agent opens with it.

A confidence threshold that hands off early

The bot now escalates when its own confidence is low and when the issue type has poor historical resolution, rather than after three failed attempts. Escalating sooner reduced containment and increased resolution, which is the trade the old metric would have punished.

Human-in-the-loop, designed rather than assumed

"There's a human in the loop" was already true before I arrived, in the sense that a human eventually appeared. That is not a design; it is a fallback. Making it real meant deciding three things explicitly:

The third row is the one that compounds. Agents were already producing the single most valuable asset in the system: correct resolutions of exactly the cases the bot got wrong, and the company was discarding all of it. Routing those corrections back turned the support team from the bot's cleanup crew into the source of its improvement, which also did more for agent trust than any amount of internal communication would have.

Two models on the irreversible actions

Credits, cancellations and anything else the customer cannot undo now go through a second model from a different vendor before execution. It sees the same conversation and answers one narrow question: does this action follow from what the customer asked for? Agreement proceeds. Disagreement escalates to a human, with both readings shown.

This catches a specific and nasty failure that single-model confidence never will: the model that has misunderstood the request is generally confident about its misunderstanding. A second model, primed differently and with no stake in the first one's reasoning, disagrees. Roughly 2% of actions get held. About a third of those turn out to be genuine misreads, a small number that would otherwise be refunds issued against the wrong invoice, and subscriptions cancelled by people asking how to pause one.

Cancellations: stop obstructing

The bot cancels. The retention offer is made once, plainly, and taken or not. Cancellation contacts stopped generating repeat contacts and angry reviews, and save rate moved by less than a point.

The number leadership reports went down, from 42 to 31. The business improved substantially. Getting a board comfortable with that requires explaining the old number was never real, which is a conversation worth having exactly once, early, rather than discovering it during a churn review.