Teach Your AI Agent How to Improve with Coaching

What is Coaching?

Coaching lets you provide feedback on what your AI Agent says, thinks, and does - enabling it to learn from past behaviour and improve. Coaching allows the AI Agent to create memories of example conversations and adjust performance in the future.

How to create good Coaching?

1. Identify AI Agent behavior that you'd like to see improved.

   When viewing conversation transcripts in the Conversations view, you can now provide feedback on certain responses and decisions made by the AI Agent. Review customer conversations to determine where you'd like to see your AI Agent's behavior improve in the future.

tip

The source conversation you use when providing Coaching feedback is important for how the AI Agent creates a memory of when to apply this feedback in the future. Coaching can be applied to real customer conversations or to your own test conversations, but be sure that the conversation is realistic and reflective of the types of situations where you'd want your feedback considered in the future.

From the conversation view, click on the 'Provide coaching' icon when hovering over a message or reasoning behavior in the transcript to provide coaching feedback.

tip

If you want to see the AI Agent ask a specific clarifying question or reframe their response to better match your expectations, coach on a problematic message sent by the AI Agent. If you want to see the AI Agent alter their decision making to better match your expectations, coach on a problematic reasoning step in the conversation log.

2. Specify the situations when this feedback should be considered.

   After you click Provide Coaching, specify the types of future conversation intents that your feedback should apply to. This helps your AI Agent understand when to recall your coaching feedback in the future. Your AI Agent will automatically suggest the situations where your AI Agent may apply your feedback (based on its analysis of the messages within the conversation) and populate this intent in the 'When replying to...' field. However, you may want to refine the scenarios when this feedback should be considered.

tip

The 'When replying to...' intent field should be as specific as possible and should emphasize only a singular scenario at a time. If you want your Coaching to be used for multiple different types of scenarios, create multiple pieces of Coaching rather than an overly complex intent.

3. Provide feedback on your AI Agent's response.

   Select the appropriate behavior that your AI Agent should use at this point in the conversation depending on whether you want your AI Agent to:

   • Send a message: Adjust it's response for clarity or highlight specific context that may have been missed (Note: for 'Send a message' type coaching, you can use the text input to 'Coach your AI agent on how to respond'. Here you have 500 characters to provide specific instructions on what the AI agent should say differently in plain language).
   • Handoff: Direct your AI Agent to use it's handoff logic to escalate future conversations
   • Use knowledge base: Select a specific article from your Knowledge Base for your AI Agent to reference when responding to similar inquiries.
   • Run an action: Select a specific action to be executed (Note: requires at least one action to be configured and active).
   • Follow a process: Select a specific process to be followed (Note: requires at least one process to be configured and active).

tip

Think of your Coaching as instructions for the AI Agent to consider in future conversations. For 'Send a message' type coaching specifically, keep your instructions simple and avoid providing instructions that are conditional on hypothetical context that your AI Agent won't have (i.e. avoid the use of multi-step instructions)

4. Validate your feedback.

   After you’ve added your feedback, click Validate to observe how the AI Agent would have responded differently with your new feedback in mind. If you don’t like the change, you can always edit your feedback and validate again.

   Once you’re satisfied with the result, click Save to add it to your AI Agent’s memory.

   Note: today you can only observe exactly how the AI Agent would change it's response when Validating 'Send a message' type coaching; if you want to observe the effect from other coaching behaviors, simply save the coaching and recreate the scenario with a new test conversation.

   

5. Monitor and observe the impact your Coaching has on new conversations.

   From the Coaching page, you can modify Coaching in your system and review conversations where your AI Agent considered your feedback. You can use this page to edit, delete, view source conversation where the coaching was first applied, and view a filtered search of additional new conversations that have taken this coaching into account.

   Note: today you can only observe the conversations where Coaching was applied but soon this table will display additional context on your Coaching's impact on AR and other metrics.

   

   In addition, you can now filter and view conversations from the conversation log for 'Coaching applied' to see conversations where your past coaching has been used. Within a conversation where coaching has been used, you can see where it was applied from within the conversation log.

Custom Instructions vs. Coaching

Custom instructions work best when used for providing general instructions that you need your AI Agent to consider with every single conversation (i.e. "Always end a conversation by asking the user if they have any more questions").

Coaching works best when used for defining specific changes in your AI Agent's past behaviour based on the unique context of only some conversations (i.e. "Next time someone asks about our shipping policy, you should first confirm if they are located in the US or Canada").

This means that your AI Agent may have some saved contextual-type Custom Instructions that may be better suited as Coaching.

Recreating Contextual Custom Instructions as Coaching

You can expect to improve your AI Agent's performance by replacing specific and contextual Custom Instructions with Coaching.

1. Identify Contextual Custom Instructions.

   Go to the Custom Instructions tab under AI Agent profile > Persona. You'll notice some of your saved Custom Instructions have been labeled as highly contextual and may be better suited as Coaching.

   

2. Deactivate and test Custom Instructions to see if any can be permanently deleted.

   Our system is always improving and optimizing for performance and some of your old Guidance may no longer be necessary.

   To check, temporarily deactivate an old Custom Instruction and create a new test conversation that's representative of a typical customer interaction.

   • If the test works well with the Custom Instruction deactivated, it's safe to delete it. It's no longer necessary and its removal will simplify your AI Agent's job.

   • If the test results in issues or undesirable AI Agent behavior, end the conversation and head to the Conversations view to provide Coaching.

3. Find a relevant example conversation to coach OR coach on your test conversation

   Coaching works by using a past conversation as an example of similar situations where you'd want your feedback applied in the future.

   Within the Conversations view, there are two ways to provide Coaching:

   • Coach directly on the test conversation you just created in step 2

   • Find a different, relevant example conversation by using the search field and keyword searching. If you're searching for a real-world example, a good starting point is to use the title of the Guidance you are looking to replace. As a general rule of thumb, ask yourself - is this an example of a conversation where I would want my feedback applied? If you can't find a suitable example, temporarily deactivate the Guidance you want to replace and create a new test conversation to mimic the situation you want to address.

   Note: you can only apply Coaching to conversations that occurred after September 18, 2024.

4. Pinpoint the behavior to change.

   Find the point in the conversation where you'd want your feedback to be applied. For example:

   • Did the AI Agent handoff where you'd want it to send a message instead?

   • Did the AI Agent search an unsuitable article from your Knowledge Base?

   • Did the AI Agent use wording that you think can be improved?

   Once you've identified the point in the conversation that has the behavior you want to change, click on the 'Provide coaching' button

5. Create and save Coaching.

   Provide Coaching that matches the Guidance being replaced. Be sure to keep your feedback framed as an instruction to the AI Agent for optimal adherence.

   Validate and save your Coaching.

6. Test.

   Go back to the Custom Instructions tab and confirm the old Custom Instruction is deactivated.

   Create a final test conversation for this scenario to verify your Coaching works as intended and confirm the old Custom Instruction is no longer needed.

7. Remove the old Custom Instruction.

   Delete the old Custom Instruction that you just replaced to improve performance.

---

Have any questions? Contact your Ada team, or email us at [email protected].