Resolution rate optimization

Overview

Measuring your AI Agent’s success is a challenging task. It’s impossible to go into every conversation that customers have with your AI Agent, and with all of the different reports and metrics you can look at, it can be hard to know which ones to focus on.

The automation resolution metric (AR) is not only a measure of whether your AI Agent was able to automatically resolve a customer’s inquiry without handing them off to human support, but also whether it was able to successfully address the reason why your customer came to your AI Agent in the first place. Your AI Agent uses AI language understanding to assess both customers’ inquiries and its own responses to figure out whether a successful automatic resolution did or didn’t happen.

Why automated resolution?

Historically, metrics like the containment rate have come close to measuring success, but they couldn’t provide a complete picture. The containment rate tells you which proportion of conversations that customers had with your AI Agent ended without being handed off to a human agent. But without context, this metric can only tell you so much. Can you differentiate between a customer who ended a conversation with your AI Agent because they were satisfied, as opposed to one who got frustrated and gave up?

While containment can be a useful metric to get a quick glance of the proportion of AI Agent conversations that didn’t escalate to a human agent, automated resolution takes it a step farther. By measuring the success of those conversations and the content they contain, you can get a much better idea of how helpful your AI Agent content really is.

Limitations

Automated resolution classification has the following constraints:

• Before February 24, 2025, only knowledge content was used to assess conversation accuracy and predict whether a user’s issues were resolved. Processes, Actions, Coaching, and Company Descriptions were not yet considered.
• Before July 31, 2024, AI Agents only assessed a statistical sample of conversations for automated resolutions. As a result, conversations that happened before this date may not have classifications.

Use cases

Automated resolution data helps you understand and improve your AI Agent’s performance.

• Measure AI Agent effectiveness: Track whether your AI Agent successfully addresses end user inquiries without requiring handoff to human support.
• Identify content gaps: Find patterns in unresolved conversations to discover missing information in your knowledge base.
• Validate conversation quality: Differentiate between end users who were satisfied and those who gave up, providing deeper insight than containment rate alone.
• Drive continuous improvement: Provide feedback on classifications to improve the model and refine your AI Agent’s content over time.

Capabilities & configuration

Understanding how automated resolution works helps you interpret and act on your AI Agent’s performance data.

How classification works

Your AI Agent analyzes each conversation to understand both the customer’s intent and the AI Agent’s response. Based on that analysis, it then assigns a classification of either Resolved or Not Resolved to each conversation.

For a conversation to be considered automatically resolved, the conversation must be:

• Relevant: The AI Agent effectively understood the customer’s inquiry and provided directly related information or assistance.

• Accurate: The AI Agent provided correct, up-to-date information.

• Safe: The AI Agent interacts with the customer in a respectful manner and avoided engaging in topics that caused danger or harm.

• Contained: The AI Agent addressed the customer’s inquiry without having to hand them off to a human agent.

To get an overview of what percentage of conversations were automatically resolved, you can use the Automated Resolution Rate report.

Conversation lifecycle

Understanding when conversations are assessed helps you interpret resolution data accurately.

For contained conversations, the AI Agent will assess a conversation for automated resolution after the conversation ends.

Conversations may end immediately if:

• A customer clicks End Chat on web chat
• A customer hangs up during a voice call, or
• A human agent ends an escalated conversation.

Idle conversations will automatically be ended after 24 hours in web chat and social channels, and 72 hours in email.

For escalated conversations, the AI Agent marks the conversation as Not Resolved immediately following the handoff.

Quick start

Review your AI Agent’s automated resolution performance in a few steps.

To view automated resolution data:

For detailed instructions on providing feedback, see Implementation & usage.

Implementation & usage

You can read through individual conversations to see how customers interacted with your AI Agent. If you disagree with how your AI Agent automatically classified a conversation, you can provide feedback.

To provide feedback on a classification:

1. Open a conversation. You can do this two ways: from the Automated Resolution Rate report, or from the Convos view.

   • From the Automated Resolution Rate report

     If you’re looking at the Automated Resolution Rate report, scroll down to the Conversations section. Click a conversation to read the entire transcript.

   • From the Convos view

     On the Ada dashboard, go to Convos. Optionally, you can use the AR Classification filter to narrow down conversations by selecting Resolved or Not Resolved to look through those conversations.

     Select a conversation in the conversation library on the left side of the screen to read through the entire transcript.

2. If the sidebar on the right side of your page is collapsed, click Details to expand it. There, you can read through your AI Agent’s automatic assessment of the conversation:

   Inquiry Summary

   An automatically generated summary of what the customer wanted to accomplish. For example, The customer wanted to know how to temporarily disable auto deposits.

   Classification

   The classification of either Resolved or Not Resolved that your AI Agent assigned to the conversation.

   Reason for Classification

   Your AI Agent’s understanding of what happened in the conversation, which led it to the classification that it assigned. For example, The AI Agent provided a detailed step-by-step guide on how to disable auto deposits, both on the mobile app and the web version.

   Take a second to understand the reasoning that your AI Agent used to make its classification, so you can give more targeted feedback.

3. Beside Agree?, click Yes or No.

   If you clicked No, enter some additional details about what your AI Agent got wrong in its assessment.

4. Click Submit.

Best practices

Automated resolution data can help you find gaps in your AI Agent’s content that you can improve.

While going through your automatic resolution data, ask yourself:

• What kinds of patterns am I seeing?

  For example, let’s say there are a lot of unresolved conversations caused by your AI Agent being unable to understand what your customer was asking for. Are there improvements you can make to your knowledge base content to improve recognition?

• Is there additional information I can add?

  If you see that multiple unresolved conversations relate to the same topic, consider adding that information to your knowledge base to address it.

Keep in mind, it’s not a realistic goal to get to 100% automated resolutions in your AI Agent. Some customers ask for handoffs immediately, and others might ask inquiries that are off-topic or even abusive, and you can’t expect your AI Agent to resolve those queries.

Related features

Explore additional optimization and reporting options.

• Conversations: View and manage AI Agent conversations.
• Performance reports: Track automated resolution rate and other metrics.
• Knowledge: Improve your AI Agent’s content to increase resolution rates.
• Coaching: Refine your AI Agent’s responses based on conversation insights.
• Handoffs: Configure escalation paths for unresolved conversations.