What a huge year it was for bots! A term that was barely on the radar became a sizzling hot trend. Facebook kicked off the mania by opening its Messenger platform and now has 33,000 bots. Competing platforms have high numbers too: 19,000 for Kik and 10,000 for Line. It looks like 2017 will continue the momentum.
But after the hype comes the backlash. Too many expectations were placed on this idea, as I wrote in ChatBot Backlash Boogie Woogie (which includes a list of 4 specific expectations and why they were misguided.)
Now that we’re past the initial crush, it’s time for bots to start the slow climb to a mature technology. One of the clearest sign-posts along that climb is the metrics that are reported.
How Do We Measure Success?
So we definitely have quantity. But do we have quality? In other words, are these bots accomplishing something?
An Important Step Forward from Facebook
In November, Facebook took us a step closer to answering that question. It announced that its free analytics platform would be extended to cover chatbots on Messenger.
As the leading bot platform, what Facebook does sets the tone for everyone else. It has, after all a billion monthly active users.
Right now, platforms are boasting “we have 30,000 bots” and companies are boasting “our bot handled 100k chats last month”. These aren’t bad numbers to report; they definitely signal progress. But what I’m really watching for is companies that start reporting success metrics rather than usage metrics.
The Old World Can Teach the New World
You might say, “Isn’t it too soon to already define metrics for a technology that is so new?”
Not if we borrow from the world of customer service analytics. There we have decades of experience and wisdom around defining and using metrics. (See our Call Center Metrics Pitfalls whitepaper for a good starting point.)
The metric that springs first to mind is “containment rate”. It answers the question: “Did the bot fully eliminate the need for an agent conversation (or other channel)?” That’s the main outcome we’re hoping for, right?
It’s not a coincidence that “containment rate” is often the main metric used to grade IVR implementation. In many ways, it’s useful to think of chatbots as the modern-day successor to the IVR. Both offer front-line, automated interaction, in an attempt to resolve issues before escalating to a live agent. When you look at it in those terms, it’s easier to see through the hype and focus on what real success looks like.