Tools

compare_changeset_metrics

Compares a change set’s performance against baseline over a date range in one call.

Baseline is the set of conversations not attributed to any change set. Your live AI Agent handles them. The tool returns baseline and change set side-by-side for each metric, with the absolute and relative difference.

Read-only. Use list_agent_changesets to find a change set ID, and edit_agent_behavior to stage, promote, or revert changes.

Example prompts

  • “How does change set abc123 compare to baseline over the last 14 days?”
  • “Did the returns-policy change improve resolution rate?”
  • “Compare my staged change against baseline for September.”

Parameters

ParameterTypeRequiredDescription
changeset_idstringYesThe change set to compare against baseline.
start_datestringYesInclusive start of the window in YYYY-MM-DD format.
end_datestringYesInclusive end of the window in YYYY-MM-DD format.
time_zonestringNoIANA time zone used to read start_date and end_date. Defaults to UTC.

Metrics returned

metric_typeDescription
containment_ratePercentage (0–100) of conversations not escalated to a human agent.
resolution_ratePercentage (0–100) of automatically resolved conversations.
csat_ratePercentage (0–100) customer satisfaction.
custom_metric_pass_ratePercentage (0–100) of conversations that passed a custom metric. One row per enabled custom metric, each with metric_id and metric_name.

Custom metrics need the custom metrics feature enabled for your workspace.

Response

Returns one object with the change set ID, the date range, per-cohort volume, and a list of metric comparisons.

FieldDescription
changeset_idThe change set that was compared.
date_rangeThe start and end dates the comparison covered.
cohortsConversation volume for each cohort. baseline.volume and changeset.volume come from opened conversation volume.
metricsOne entry per metric. See the fields below.
truncated_custom_metricsNumber of custom metrics left out when the workspace has more than the per-call cap. null when nothing was left out.

Each entry in metrics has these fields:

FieldDescription
metric_typeThe metric name, for example containment_rate.
baselineThe metric value for the baseline cohort.
changesetThe metric value for the change set cohort.
deltachangeset minus baseline.
relative_deltaThe change relative to baseline, as a percentage (delta / baseline × 100). A value of 4.6 means the change set is 4.6% above baseline.
metric_idThe custom metric ID. Set on custom-metric rows, null on other rows.
metric_nameThe custom metric name. Set on custom-metric rows, null on other rows.
breakdownPer-cohort evaluation counts, {"baseline": {...}, "changeset": {...}}. Each side holds pass_count, fail_count, null_count, and total_evaluated. Set on custom-metric rows, null on other rows.

All rate values are percentages (0–100) rounded to 1 decimal place.

A cohort with no conversations returns null for that side, and delta and relative_delta are then null. A zero-volume cohort has no real rate, so the tool reports null rather than a fabricated 0.

Why use this over get_ada_metric

get_ada_metric returns one metric for one cohort. To compare a change set against baseline with it, you call it twice per metric and align the results yourself. This tool returns every metric for both cohorts, with the difference computed, in one call.