Containment rate is a vanity metric

Containment rate — the share of conversations a bot handles without escalating to a human — is the first number most support teams put on the wall. It’s easy to compute, it moves when you tune the bot, and it maps neatly onto cost per contact.

It’s also the metric most likely to make a deteriorating customer experience look like a win.

What containment actually counts

Containment measures conversations that didn’t reach an agent. It does not measure conversations that ended well. Those are very different populations, and containment merges them:

• The customer whose question the bot answered correctly.
• The customer who gave up and closed the window.
• The customer who couldn’t find the path to a human and went to a review site instead.
• The customer who got a confident, wrong answer and left satisfied — until it mattered.

All four are contained. Three of them are problems, and one of them is a problem you won’t hear about until it’s expensive.

This is why containment can improve at exactly the moment experience degrades. Make escalation harder and containment rises. Shorten the bot’s willingness to say “I don’t know” and containment rises again. The dashboard rewards the behaviour you’d never choose deliberately.

The blind spot is structural

The data you need to tell those four cases apart is mostly not in the conversation log. A transcript shows what was said, not whether the problem went away. So teams fall back on proxies — session length, message count, a thumbs-up widget almost nobody clicks — and those proxies are weakly correlated with resolution at best.

Meanwhile the clearest evidence is sitting one system over: whether the customer came back.

Resolution is a question about what happened next

A more honest definition: a conversation is resolved if the customer did not need to ask again.

Operationally, that means looking at a window after the conversation ends — 24 hours, 72 hours, whatever matches your product — and asking whether the same customer opened another contact about the same thing, on any channel.

That one change reframes the whole measurement:

• Repeat contact rate catches the customer who gave up and came back by email.
• Escalation-after-containment catches the bot that deflected and the human who paid for it.
• Abandonment mid-conversation catches the customer who left without resolution and without complaint — the group containment hides most effectively.

None of these require new instrumentation beyond linking conversations to a customer identity over time. Most teams already have the data and simply aren’t joining it up.

Where the useful signal lives

Once you measure resolution rather than deflection, the interesting question stops being how many did the bot handle and becomes which intents does it handle well.

Bot performance is almost never uniform. It’s usually excellent on a handful of high-volume, low-ambiguity intents — order status, opening hours, password reset — and poor on anything involving context, exceptions or emotion. A single containment figure averages those together and tells you nothing actionable.

Broken out by intent, the same data tells you precisely where to invest:

• High volume, high resolution → protect it; this is the automation paying for itself.
• High volume, low resolution → the top of the fix list.
• Low volume, low resolution → route it straight to a human and stop pretending.

That last category is where a lot of goodwill quietly disappears. Rare, complicated, often emotionally charged — and precisely the conversations where a bot’s confident guess does most damage.

What to put on the wall instead

If you replace one number, replace containment with resolution within the window, segmented by intent, with abandonment reported alongside it.

It’s a less flattering number. It will drop when you first measure it honestly, and you should expect that. It has one decisive advantage over containment: when it goes up, something actually got better for a customer.

Cost per contact is worth tracking. But a bot optimised for deflection eventually produces a support function that’s cheap, efficient, and quietly losing people.

---

ChatMetric is a conversational AI analytics platform that helps support teams see what their bots are really doing. More at chatmetric.net.