Bots are a great tool to have. Enterprises have been flaunting them for every function.
- Customers use bots to search for products, get answers from FAQs, perform self-service, provide feedback, to escalate product issues to human agents, and many more
- Employees leverage them to automate repetitive actions, fetch information, generate reports, update systems of record, and perform complex workflow tasks etc.,
- Then there are specialized bots for respective functions – IT Help Desk, Customer Support, Sales, Marketing, HR etc.,
- There are also universal bots that employees can invoke to talk to multiple specialized bots to get tasks done
Besides, enterprise bots live in various channels of communication – email, SMS, messengers, IVR, mobile/web apps, voice assistants etc., and converse in multiple languages with people, systems and things, to make employees productive and customers, successful.
But as bots proliferate and impact many enterprise functions, the focus has shifted to leveraging them in the right manner, with the right kind of capabilities. More is not merrier if we cannot manage them properly.
In a recent blog we explained why enterprises, given their unique requirements, need platforms that ensure end-to-end bot life cycle management.
A critical aspect of this life cycle management is metrics. Business and IT managers need to get a holistic experience of a bot’s performance, and at a single click, to refine its efficiency through analytics-driven insights.
As not all bots respond alike, they need to be continuously analyzed and fine-tuned for performance and productivity. After all, what gets measured, gets managed better.
To scale bots efficiently across the enterprise and to increase adoption, usage, performance, task execution, natural language understanding and response generation, user intent recognition and entities extraction, back-end systems performance, communication channels integration, human agent hand-offs etc.
Also Read: The Statistical Side of Chatbots |
Basic metrics part of this analysis includes: conversations the bots are having, users that are engaging the bots, channels the bots are working in, the types of tasks the bots are executing, time taken to perform the tasks, successfully executed tasks, tasks that failed to execute, reasons for failure, the dialog paths, the nodes where the conversations broke, training opportunities identified, sentiments analyzed etc.
Analytics scale conversations and spur productivity
Chatbot analytics support enterprise scalability, universal security, real-time training, increased adoption, and improved ROI through incremental employee productivity and customer success. Some metrics include: tasks performed, alert notifications sent, number of sessions, average session time, average sessions per user, average chats per session etc.
Areas where analytics are applied
Tasks and notifications
Also Read: Analytics in Conversational AI |
- On-demand tasks: An increase in the uptake of tasks is a sign of improving adoption of the bots, an indicator of incremental productivity. Task variance is a measure of bot versatility.
- Notifications sent: An increase in notifications also reflects bot usage and task efficiency.
Sessions and chats
The length of user-bot sessions and the number of chats per session is a direct measure of bot adoption, intent recognition, and task completion. Metrics include: average session time, average sessions per user, average chats per session etc.
Users and groups
Users maintain roles and associated permissions, and sometimes belong to groups with restricted permissions or may have accounts with full privileges across the platform. Other information includes gender, location, language, function etc. Bot usage is tracked across the layers of the people organization to analyze user retention and bot adoption.
Conversation channels and communication experiences
Users can come in from any communication channel – messengers, email, web, etc. Every channel provides a unique experience, impacting the performance of the bot. As conversations move across channels and between groups, it becomes increasingly complex to trace paths and fix breaks. The conversation volleys and channel transitions are tagged and tracked for contextual troubleshooting.
Dialog paths and node breaks
Conversation paths and branches are tracked to detect node breaks, where tasks failed to execute. Reasons could include misunderstood user intents, broken integrations, or other system defects. The context objects containing the bot’s traversal provides clues to improve conversation design and intent recognition capabilities.
Also Read: Bots Make Data-Driven Analytics and Insights a Reality for All Decision Makers |
Intent recognition and task failures
Incorrectly identified user intents are collated and separated for further training using machine learning tools. Continuous retraining minimizes false positives and expands the bots’ learning from real-life experiences. Chat transcripts (and JSON responses) are obtained to identify the exact nodes where the conversations failed.
Some reasons for task failures include: task aborted by the user, chat interface refreshed, authorization attempt failure, script / service failure, incorrect entity value provided, chat transferred to human agent etc.
Human agent hand-offs
Bots identify user utterances using NLP techniques and score words or phrases mapped to emotions, to analyze user sentiments. The bot automatically brings in a human agent into the conversation when it does not have the relevant information or when a user expresses negative emotions.
Transfer to a human agent can also be pre-designed in a dialog builder based on conditional utterances. The number of agent transfers are analyzed to fix the conversation design or identify bot-training opportunities with new utterances. Chat transcripts are obtained and analyzed in real time to troubleshoot the reasons for failure, with the final goal of enabling customer self-service.
Enabling analytics reporting
All the above data points, transactions and behaviors must be presented as dynamic graphs for easy visualization and drill-down decision making. Conversations must be suitably tagged to be later retrieved for contextual analysis by semantic searching. APIs must be open and available for data extraction and custom reporting using third party tools or analytics engines.
A chatbot platform with built-in analytics, that can be accessed in real-time, is a critical success factor for the speedy bot deployment and measurable return on investment.
Explore conversational technology at: kore.ai
Build your own bots with built-in analytics at: developer.kore.ai
