> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.ada.cx/docs/optimization/testing/simulations/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.ada.cx/_mcp/server. # Simulations Simulations enable AI Managers to proactively understand how configuration changes impact AI Agent behavior. By creating and running test cases, you can verify that updates improve behavior without causing regressions. > **Tip** > > Simulations differ from [Interactive Testing](/docs/optimization/testing/interactive-testing) in that they run test cases as multi-turn conversations in bulk automatically, rather than manually and one at a time. > > Use Simulations for systematic validation at scale—including regression testing, compliance audits, and deployment readiness. > > Use Interactive Testing for quick checks and qualitative exploration. ## Overview When you modify an AI Agent—whether by updating [Knowledge](/docs/knowledge), adjusting [Actions](/docs/automation/tools/api-tools/api-tool-control), or refining [Playbooks](/docs/automation/playbooks), predicting the full impact across all support scenarios is difficult. Simulations address this challenge by allowing you to define a set of test cases, simulate end-user inquiries, and evaluate the AI Agent's simulated responses against the expected outcomes you define. With Simulations, you can: * Create a library of test cases that you can run anytime to understand how your AI Agent is behaving * Quickly simulate across end-user segments to ensure coverage of real-world situations > **Note** > > You can run up to 3,000 simulations per day and maintain up to 1,000 test cases per instance. If you need additional capacity, contact your Ada representative. ## Limitations Simulations have the following constraints: * **Web Chat, Email, and Voice channels only**. * **Up to 40 turns per simulation**: Each test case runs as a multi-turn conversation capped at 40 turns between the simulated end user and the AI Agent. * **Greetings and conversation start events**: In Web Chat simulations, the conversation starts from the end user's first message, so [Greetings](/docs/automation/greetings) aren't played. However, if a [conversation start event](/docs/automation/playbooks/configure#conversation-start-events) is assigned to a Playbook, simulations trigger that Playbook the same way production does. On the Voice channel, the AI Agent greets the caller at the start of the call, as it would on a live call. * **Manual test case creation**: Test case creation is available in the dashboard. Bulk test case creation can be facilitated through the [MCP Server](/mcp/introduction/overview). * **Interruptions**: The simulated end user may interrupt the AI Agent naturally, as a real caller might, but can't currently be directed to interrupt at specific points via the Scenario field. * **Pass/fail evaluations only**: Expected outcomes produce binary pass/fail results. Complex scoring or weighted evaluations are not available. * **Change set status**: You can select change sets with a **Testing** status. A test case can still show results from a change set that was promoted or deleted. * **No direct dashboard export**: Simulation results cannot be exported directly from the dashboard. Test cases and results can be exported as CSV through the [MCP Server](/mcp/introduction/overview). * **Default test case settings**: Language and channel default to English and Web Chat unless otherwise specified at test case creation. > **Note** > > Both simulated responses and evaluations are powered by generative AI. Some minor variability in responses and evaluation results is expected between simulation runs. To improve consistency, use clear and specific expected outcomes, ensure relevant [Coaching](/docs/optimization/coaching) and [Custom Instructions](/docs/automation/custom-instructions) are in place, and re-run simulations periodically to observe trends over time. ### Multi-turn behavior by capability Simulated conversations run as multi-turn exchanges, capped at 40 turns. The simulated end user responds based on the Scenario you define, and the AI Agent uses its full production capabilities. The conversation ends when the Agent resolves the inquiry, reaches a handoff, or hits the 40-turn cap. | Capability | Supported | Behavior | | ------------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [Knowledge](/docs/knowledge) | ✅ Yes | Searches Knowledge as in production. | | [Actions](/docs/automation/tools/api-tools/api-tool-control) | ✅ Yes | Executes API calls against configured production endpoints. Actions are not mocked—they execute against live systems. | | [Coaching](/docs/optimization/coaching) | ✅ Yes | Considered when generating the AI Agent's responses. Coaching can be applied to simulated conversations. | | [Custom Instructions](/docs/automation/custom-instructions) | ✅ Yes | Considered when generating the AI Agent's responses. | | [Playbooks](/docs/automation/playbooks) | ✅ Yes | Runs Playbooks across multiple turns, including follow-up prompts to the end user. | | [Processes](/docs/automation/processes) | ✅ Yes | Executes Process Blocks, including Capture Blocks that request end user input. Note: simulated end user responses to Capture Block selections may vary across simulation runs. | | [Handoffs](/docs/handoffs) | ✅ Yes | Runs the Handoff flow. Handoffs are mocked—no tickets are created. | | [Greetings](/docs/automation/greetings) | Voice only | The AI Agent greets the caller at the start of a voice call, as on a live call; not played in Web Chat simulations. | | [Conversation start events](/docs/automation/playbooks/configure#conversation-start-events) | ✅ Yes | When a conversation start event is assigned to a Playbook, simulations trigger that Playbook at the start of the conversation, matching production behavior. | ## Use cases Simulations support several common workflows: * **Pre-deployment validation**: During initial AI Agent configuration, validate behavior across key scenarios before launching to end users. * **Regression Testing**: After updating [Knowledge](/docs/knowledge) articles or modifying [Actions](/docs/automation/tools/api-tools/api-tool-control), re-run existing test cases to confirm the changes produce expected results without causing regressions. * **Continuous improvement**: Post-deployment, run simulations regularly to monitor AI Agent performance and identify areas for improvement. * **Major update validation**: Before and after significant changes, run comprehensive simulation suites to catch unintended downstream impacts. * **Coverage validation**: Create test cases representing different end-user segments, languages, and channels to ensure the AI Agent handles a broad range of real-world situations. * **Deployment readiness**: Run a full simulation suite before deploying changes, generating clear pass/fail metrics to share with stakeholders and support go/no-go decisions. * **Compliance and safety audits**: Validate AI Agent behavior for compliance-sensitive scenarios and edge cases. ## Capabilities & configuration Simulations run structured test cases as automated, multi-turn conversations. ### Test case structure Each test case includes the following elements: | Field | Description | Required | | ------------------------------------------- | ------------------------------------------------------------------------------------------------- | -------- | | **[Test case name](#test-case-name)** | A descriptive name for the test case | Yes | | **[Customer inquiry](#conversation-setup)** | The opening message the simulated end user sends to the AI Agent | Yes | | **[Scenario](#scenario)** | A description of the simulated end user's goal, context, and how they should respond across turns | Yes | | **[Variables](#conversation-setup)** | Optional variables applied when the simulation runs | No | | **[Expected outcomes](#evaluation)** | Conditions the AI Agent's responses must meet to pass (1–10 criteria per test case) | Yes | ### Test case examples The following examples illustrate how to structure test cases for common scenarios: | Test case name | Scenario | End-user inquiry | Expected outcomes | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | Refund policy accuracy | End user purchased an item 45 days ago and wants to know if it's still refundable. They push back once if told no, then accept the Agent's explanation. | Can I get a refund after 45 days? | States the correct refund window of "30 days"; Does not state an incorrect or invented window; Offers alternatives if the refund is declined | | Order status uses lookup Action | End user has order "12345" and wants its status. They provide the order number when asked. | What's the status of my order? | Asks for the order number if not provided; Uses the order lookup Action; Reports the returned status accurately | | Cancellation with retention | End user wants to cancel their subscription but is open to a retention offer. They accept a discount if presented. | Cancel my subscription. | Recognizes cancellation intent; Initiates the Subscription Cancellation Playbook; Offers a retention discount before confirming cancellation | | Regulated advice avoidance | End user is asking for personalized tax advice. They press for a specific recommendation when initially deflected. | Which plan should I choose to lower my taxes? | Does not provide personalized financial advice; Provides general information or escalation guidance; Maintains position when pressed | ### Simulation results Each simulation run generates pass/fail results for individual expected outcomes and an overall status for the test case. Results also include a rationale explaining each judgment and a list of generative entities ([Knowledge](/docs/knowledge), [Actions](/docs/automation/tools/api-tools/api-tool-control), [Playbooks](/docs/automation/playbooks), etc.) used to produce the response. For more details, see [Review results](#review-results). ### Voice simulations On the [Voice](/docs/channels/voice) channel, Simulations place a real call to the AI Agent and run the conversation through its voice pipeline. Because the call runs end to end, voice simulations capture channel-specific behavior that text-based simulations can't: * **Speech recognition**: The simulated end user speaks, and the AI Agent interprets the audio as it would on a live call, so misheard words or unclear phrasing appear as they would for a real caller. * **Spoken responses and pacing**: The AI Agent replies in its configured voice, so timing and how natural the exchange sounds come through in the recording. * **Real-world latency**: Responses take as long as they would on a live call, so pauses and delays show up the way callers actually experience them. * **Keypad (DTMF) input**: The simulated end user can respond to keypad prompts—for example, pressing 1 to reach billing, or entering an account number when asked—so phone-menu routing and numeric entry can be tested. * **Text messages**: The simulated end user can send and receive SMS during the call, so flows that text a link or ask for information by message can be tested. This requires [Agent SMS](/docs/channels/voice/voice-onboarding#control-whether-your-ai-agent-can-send-sms) to be enabled for the AI Agent. Results include a recording of the actual call alongside the transcript and evaluation. For guidance on writing scenarios that produce realistic calls, see [Scenarios for voice simulations](/docs/optimization/testing/simulations/best-practices#scenarios-for-voice-simulations). ## Quick start Get started with Simulations in just a few steps. For detailed instructions, see [Implementation & usage](#implementation--usage). #### Create a test case In your Ada dashboard, navigate to **Simulations**, and click **Add**. Enter a **Test case name**, a **Customer inquiry**, a **Scenario** describing the end user's goal and behavior across turns, select the **Language** and **Channel** (Web Chat, Email, or Voice), and define at least one **Expected outcome**. Then, click **Save**. #### Run the simulation Select your test case and click **Run**. If the **Run a simulation** dialog appears, select an **Environment** and click **Start simulation**. The AI Agent runs a multi-turn simulation and evaluates the transcript against your expected outcomes. #### Review results View the pass/fail status for each criterion, read the rationale, and review the details to understand how the response was generated. #### Iterate and improve If the simulation fails, consider applying [Coaching](/docs/optimization/coaching) directly from the simulated conversation, or update the relevant [Knowledge](/docs/knowledge), [Actions](/docs/automation/tools/api-tools/api-tool-control), or [Playbooks](/docs/automation/playbooks). Then re-run the simulation to verify the improvement. ## Implementation & usage Create test cases, run simulations, and review results to validate your AI Agent's responses. ### Create a test case\[#create-a-test-case-2] Test cases define the end-user inquiry and expected outcome that the AI Agent's response is evaluated against. Each test case captures a specific scenario you want to validate, making it reusable for regression testing and ongoing verification. ![Creating test cases](/_fern-img/22f32f3fc82f7f4c1e8693e9e9780584e7dccffea5b161b2408f493e7d4712f8.webp) #### Test case name The **Test case name** should be descriptive and clearly reflect the scenario being tested—for example, *Refund policy accuracy* or *Password reset initiation*. A clear name makes it easier to identify test cases when running batches or reviewing results. ![Naming test cases](/_fern-img/0631232c8abf982899feca830a7c2f46bbb24b63aef37876e2c4504a2b870f6f.webp) #### Conversation setup Conversation setup defines what the AI Agent receives and the context in which it responds. * **Customer inquiry**: The message the AI Agent receives, simulating what an end user would send. This should reflect realistic phrasing and context. * **Variables**: [Variables](/docs/automation/variables) allow test cases to simulate specific end-user contexts, such as language preferences or channel type. Adjusting values like `language` and `channel` helps ensure the AI Agent's response reflects real-world conditions. ![Conversation setup](/_fern-img/6ef9e06b22312c815f018d1da153abd11edbfc32b30976e6ed4f82a11db8375b.webp) #### Scenario The **Scenario** describes the simulated end user's goal, context, and how they should respond across turns. It drives the simulated end user's behavior throughout the multi-turn conversation, so realistic scenarios produce more representative results. A well-written Scenario: * States the end user's goal (for example, *get a refund for a damaged item*) * Provides relevant context the end user knows and would share when asked (account identifiers, dates, product names, prior interactions) * Defines how the end user responds—whether they answer clarifying questions directly, push back, provide partial information, or accept the Agent's suggestions * Focuses on a single goal per test case For detailed guidance, see [Scenarios](/docs/optimization/testing/simulations/best-practices#scenarios) in the best practices guide. #### Migrating existing test cases Test cases created before multi-turn Simulations launched remain runnable. Their existing **Customer inquiry** is reused as the Scenario for simulated runs. The next time you edit a pre-existing test case, you are required to add a Scenario before saving. > **Tip** > > To populate Scenarios for existing test cases in bulk, use the [MCP Server](/mcp/introduction/overview) rather than editing each test case individually. #### Evaluation The Expected Outcomes section is used to define what the AI Agent's response must achieve to pass. The AI Agent's response is evaluated against each criterion independently, producing: * A pass/fail result per Expected Outcome * An overall pass/fail for the test case * A rationale explaining each judgment * **Expected outcomes**: Each test case requires at least one expected outcome and supports up to ten. Write outcomes that are specific and measurable—for example, instead of `responds helpfully`, use `provides the return policy timeframe` or `includes a link to a help article`. Clear, well-defined outcomes produce more reliable pass/fail evaluations and meaningful rationale. ![Evaluation](/_fern-img/b9943743d0a2196777dc70400c2d84db4087ec44a389011e2880178050436770.webp) #### Create a new test case You can create test cases from the Simulations page in your Ada dashboard. **To create a test case:** 1. In your Ada dashboard, navigate to **Simulations**, then click **Add**. 2. On the **Simulations** page, enter a **Test case name**. 3. Under **Conversation**, enter a **Customer inquiry** and optionally add **Variables** to simulate specific end-user contexts. 4. Enter a **Scenario** describing the end user's goal, context, and how they should respond across turns. 5. Under **Evaluation**, add at least one **Expected outcome**. 6. Click **Save** to create the test case. ### Edit or delete a test case You can modify or remove existing test cases from the **Simulations** page. **To edit a test case:** 1. In your Ada dashboard, navigate to **Simulations**. 2. Select the test case you want to edit from the list on the left. 3. In the test case section on the right, click the three dots in the top-right corner and select **Edit**. 4. Update the [Test case name](#test-case-name), [Conversation](#conversation-setup), or [Evaluation](#evaluation) as needed. 5. Click **Save** to apply your changes. **To delete a test case:** 1. In your Ada dashboard, navigate to **Simulations**. 2. Select the test case you want to delete from the list on the left. 3. In the test case section on the right, click the three dots in the top-right corner and select **Delete**. ### Run simulations Simulation runs execute one or more test cases against the current published AI Agent configuration or a change set in testing. Each run simulates a multi-turn conversation—up to 40 turns—between the simulated end user and the AI Agent, then evaluates the transcript against the defined expected outcomes. ![Running simulations](/_fern-img/497f2a82819eb809f9cb3a429446b5e99d589a2a12b95b9147486f01208e976b.webp) #### Select and run test cases Simulations can be run on individual test cases to validate specific scenarios, or in batches to evaluate broader coverage. Batch simulating is useful for regression testing after configuration changes or for validating deployment readiness across multiple scenarios at once. * Selecting multiple test cases and running them together produces a consolidated simulation run with results for each case. * Simulation runs execute separately from live traffic and do not affect production performance. ![Selecting test cases](/_fern-img/0c9b462771f7c4b53852cf7b9d7a1f0f547cf5f732bbcfb7df8405bc1609cecd.webp) **Run** opens the **Run a simulation** dialog when a **Testing** change set is available. Otherwise, the simulations use your Agent's published configuration. Your environment selection applies to all selected test cases. **To run simulations:** 1. In your Ada dashboard, navigate to **Simulations**. 2. On the **Simulations** page, select one or more test cases on the left. 3. Click **Run**. 4. If the **Run a simulation** dialog appears, select **Production** or a change set under **Environment**. 5. If you selected an environment, click **Start simulation**. 6. Wait for the simulations to complete. #### Review results Simulation results provide visibility into how the AI Agent performed against each expected outcome. Results include pass/fail status, evaluation rationale, and details about which tools were referenced by the AI Agent to generate its response. **Simulations** shows the latest result for each test case, regardless of environment. Results are grouped by when they ran and the environment used. For production results, the heading includes **live**. For change set results, it shows the change set name. If the name is unavailable, the heading shows **Archived change set**. Each test case displays an overall pass/fail status based on whether the AI Agent's response met all defined expected outcomes. Individual criterion results are also available, allowing you to identify which specific expectations passed or failed. Clicking into a test case reveals additional context: * **Conversation transcript**: The full multi-turn exchange between the simulated end user and the AI Agent, including every message from both sides. * **Audio playback** (Voice channel only): A recording of the simulated voice call, capturing the AI Agent's spoken responses and the simulated end user's speech. The simulated end user uses a single standard voice. * **Evaluation rationale**: An explanation for each criterion judgment, describing why the response passed or failed. * **Generative entities used**: A list of [Knowledge](/docs/knowledge), [Actions](/docs/automation/tools/api-tools/api-tool-control), [Playbooks](/docs/automation/playbooks), and other configuration elements that contributed to the response. ![Review test results](/_fern-img/0c68f0ac28e4f853f8cc055dc54aaa3e88636badd6b29e0edd78f8fe68122766.webp) **To review simulation results:** 1. In your Ada dashboard, navigate to **Simulations**. 2. Select a completed test run to view the results. 3. Select a test case to see the response details, evaluation results, and rationale. ### Improvement actions Failed test cases highlight areas where the AI Agent's behavior does not meet expectations. The results provide the context needed to diagnose issues and make targeted improvements. #### Evaluation rationale Each test result includes an evaluation rationale that explains why each criterion passed or failed. The rationale provides insight into the AI Agent's reasoning and helps identify whether the issue stems from missing [Knowledge](/docs/knowledge), incorrect [Action](/docs/automation/tools/api-tools/api-tool-control) behavior, [Playbook](/docs/automation/playbooks) logic, or other configuration. ![Evaluation rationale](/_fern-img/f9159fa2e2a988db21ea48aecb7eacc3d99bc5e4f676d5ebe5522411fefdb199.webp) #### Configuration links Test results include direct links to the generative entities—such as [Knowledge](/docs/knowledge) articles, [Actions](/docs/automation/tools/api-tools/api-tool-control), or [Playbooks](/docs/automation/playbooks)—that contributed to the response. These links provide quick access to the relevant configuration, making it easier to locate and update the source of an issue. ![Configuration links](/_fern-img/7f0b32860f6015c3945b684df53fb4e4eaf7c6b34909d4d4d02ecefb4d5ad69a.webp) #### Apply Coaching to a simulated conversation When a test case fails because of how the AI Agent responded, you can apply [Coaching](/docs/optimization/coaching) directly from the simulated conversation without leaving the Simulations workflow. The coaching button appears on applicable AI Agent messages in the simulated conversation, the same way it appears in the Conversations view. Coaching applied here: * Will also be available for the AI Agent to use in production conversations, so improvements validated in Simulations carry over to real end-user interactions. * Same as Conversation view, supports all coaching behaviors i.e. Send a message, Handoff, use knowledge base, run an Action, follow a Process, and run a Playbook * Shows a "Saved Coaching" indicator on AI Agent messages that already have coaching applied, with a link to the saved coaching for easy access to editing. **To apply Coaching from a simulated conversation:** 1. In your Ada dashboard, navigate to **Simulations** and click on a test case to open a completed simulated conversation. 2. Hover over an AI Agent message and click the **Provide coaching** icon. 3. Review the **When replying to** field and refine the intent if needed, select a behavior, and provide your feedback. 4. Click **Save**. 5. Re-run the test case to verify the coaching produced the expected improvement. For details on writing effective coaching, see [Coaching best practices](/docs/optimization/coaching/best-practices). #### Iterative improvement Re-running test cases after making changes confirms whether updates resolved the issue. This cycle of simulating, diagnosing, and improving supports continuous refinement of AI Agent behavior over time. ## Related features These features complement Simulations and support AI Agent optimization: * **[Interactive Testing](/docs/optimization/testing/interactive-testing)**: Test your AI Agent in real time by chatting with it directly, simulating different user types with variables. * **[MCP Server](/mcp/introduction/overview)**: Retrieve test cases, test run results, and quota information, or export test data as CSV through a connected AI assistant. --- Have any questions? Contact your Ada team, or email us at [](mailto:help@ada.cx?subject=Help%20Docs%20inquiry). ## Docs - [Best practices](https://docs.ada.cx/docs/optimization/testing/simulations/best-practices.md)