Bots Haven’t Truly Arrived Until We See Containment Stats

Bots Haven’t Truly Arrived Until We See Containment StatsWhat a huge year it was for bots! A term that was barely on the radar became a sizzling hot trend. Facebook kicked off the mania by opening its Messenger platform and now has 33,000 bots. Competing platforms have high numbers too: 19,000 for Kik and 10,000 for Line. It looks like 2017 will continue the momentum.

But after the hype comes the backlash. Too many expectations were placed on this idea, as I wrote in ChatBot Backlash Boogie Woogie (which includes a list of 4 specific expectations and why they were misguided.)

Now that we’re past the initial crush, it’s time for bots to start the slow climb to a mature technology. One of the clearest sign-posts along that climb is the metrics that are reported.

The New Customer Service Lexicon Banner

How Do We Measure Success?

One of the main use cases for bots is as a new channel for customer service. We’ve seen many companies jump on board to try this out including, in the last two months, American Express and EBay.

So we definitely have quantity. But do we have quality? In other words, are these bots accomplishing something?

An Important Step Forward from Facebook

In November, Facebook took us a step closer to answering that question. It announced that its free analytics platform would be extended to cover chatbots on Messenger.


Bot Analytics


As the leading bot platform, what Facebook does sets the tone for everyone else. It has, after all a billion monthly active users.


Facebook Messenger Growth


Right now, platforms are boasting “we have 30,000 bots” and companies are boasting “our bot handled 100k chats last month”. These aren’t bad numbers to report; they definitely signal progress. But what I’m really watching for is companies that start reporting success metrics rather than usage metrics.

The Old World Can Teach the New World

You might say, “Isn’t it too soon to already define metrics for a technology that is so new?”

Not if we borrow from the world of customer service analytics. There we have decades of experience and wisdom around defining and using metrics. (See our Call Center Metrics Pitfalls whitepaper for a good starting point.)

The metric that springs first to mind is “containment rate”. It answers the question: “Did the bot fully eliminate the need for an agent conversation (or other channel)?” That’s the main outcome we’re hoping for, right?

It’s not a coincidence that “containment rate” is often the main metric used to grade IVR implementation. In many ways, it’s useful to think of chatbots as the modern-day successor to the IVR. Both offer front-line, automated interaction, in an attempt to resolve issues before escalating to a live agent. When you look at it in those terms, it’s easier to see through the hype and focus on what real success looks like.


[Whitepaper] The New Customer Service LexiconThe New Customer Service Lexicon

In this whitepaper, we’ll discuss terms that are either new to the customer service sector, or have a recently changed meaning.

Hopefully this helps to steer you in the right direction!

What’s Inside:

  • Chat vs. Messaging
  • Virtual Agent vs. Intelligent Agent
  • Virtual Queuing vs. Virtual Hold
  • Conversational Commerce
  • Plus So Much More Info!