Writing evals

The parts you author: scenario shapes, rubric questions, personas, degraded line conditions, and the metrics that carry a rubric question onto every real call.

An eval has three parts, all authored in the agent's Tests section:

a scenario: the conversation that plays;
a rubric: the questions that grade it;
a persona: the simulated caller it runs as.

See Evals for how the judge grades a run and what it costs.

The baseline eval every agent starts with

Every agent starts with one eval: Baseline, voice and persona. It runs a short, content-free call (one open question and a wrap-up) with a rubric that checks the universals every voice agent must get right:

staying in persona, not drifting into a generic assistant;
clean spoken output: no markdown, asterisks, bullets, or emoji;
no invented facts, figures, prices, or commitments;
replies that are concise and suitable for a phone call;
a polite close when the caller signals they are done.

The baseline is a starting point. Keep it, and add scenarios for the real calls your agent handles: looking up an order, booking an appointment, transferring to a human. Until an agent has its own tests the editor flags it as a readiness risk.

Scenarios

A scenario drives a simulated caller through a call so the same exchange runs the same way every time. It comes in one of three shapes.

Scripted

A flat list of caller utterances, played turn by turn. The simplest shape, and what the baseline eval above uses.

Conditional

A step-by-step caller whose next line depends on what the agent did. Each step has a trigger and the caller's reply.

trigger: when the agent asks for an order number
reply:   "it's 4-8-2-9-1"

Use this for branching calls where the right next thing to say depends on the agent's response.

Instruction

A free-text brief the simulated caller improvises within.

you are a frustrated customer whose order is late and you want a refund

Use this when you care about the behaviour, not the exact words.

Rubrics

A scenario on its own only plays the call. The rubric grades it: a list of plain-English yes/no questions a judge answers about the transcript. Each question is one thing the agent had to get right.

did the agent read the order number back to the caller before acting on it?
did the agent decline the out-of-scope request politely instead of guessing?

Writing a question that grades the same way twice

Write rubric questions as specific, checkable behaviours. A vague question ("was the call good?") grades inconsistently. A sharp one ("did the agent look up the price with a tool rather than stating it from memory?") grades the same way every time.

Personas

A persona is the simulated caller's tone, patience, and intent, written as a second-person behaviour brief.

you are a busy customer who interrupts often and gives short answers

Personas are workspace-scoped, so you write one once and reuse it across agents and scenarios. Running the same scenario against several personas is how you find out whether the agent holds up with an impatient caller, not just a polite one.

Line conditions

A scenario can simulate a degraded phone line, so you test the agent the way callers actually reach it. Four knobs are available per scenario:

Background noise. Ambient noise mixed into the caller's microphone.
Network latency. Added delay on the caller's audio, up to two seconds.
Audio quality. Line fidelity, lowered to model a poor codec or a cheap handset.
Accent intensity. How strongly the caller's accent is applied.

All default to off (clean line). Turn them up to find out whether the agent still confirms a garbled order number when the line is bad.

Metrics

A rubric question grades one scenario. Scoring turns the behaviours you care about into reusable measures that score every real call, not just test scenarios. It lives in the agent's Scoring section.

A metric is a single named measure of a call, for example "did the agent resolve the caller's task?" Metrics are defined once in the workspace catalogue and reused across agents. A language-model judge metric is a rubric question promoted into a standing measure: you write the yes/no question, and from then on it scores live calls.

Turning a metric on for an agent

Metrics are workspace-scoped but toggled per agent. A metric does nothing on an agent until it is activated there. On the Scoring, Metrics page you see every workspace metric with this agent's effective on or off state and where that state comes from:

Default on: a metric with no override scores this agent's calls.
Workspace default: turned on or off for every agent at once.
Agent override: turned on or off for this agent specifically, overriding the workspace default.

Toggle a metric on for an agent to start measuring it, or off to stop.

The success rule

A call's success verdict is computed from the critical checks rather than from a policy you author. There is no rule to write: switch the checks you care about to critical on the Scoring page and the verdict follows from them.

Seeing metrics on calls

Activated metrics appear on each transcript and roll up on the agent's overview. See Calls and analytics.

Writing evals

On this page