Evals

An eval does repeatably, and at scale, what you do by hand in the Test panel.

Evals give you three things you cannot get by retrying calls by hand:

• Repeatability. Run the same conversation again after a prompt or tool change and see whether the result held.
• Breadth. Cover the awkward cases on purpose: the caller who changes their mind, gives a wrong detail, talks over the agent, or asks something out of scope. You would never think to retry those by hand every time.
• Confidence across runs. Trust a result that holds across several runs of the same scenario, never a single pass. Five runs tell you far more than one green run.

Together they move you from "it worked when I tried it" to "it works every call".

Why a single green call proves nothing

The agent runs on a language model, so the same prompt can answer beautifully on one call and stumble on the next. The bar for a phone agent is reliability: it has to behave correctly on every call, not most calls.

One bad call in ten is too many when a caller is waiting on the line. The discipline that follows: write a failing scenario for the behaviour you want, then change the prompt or a tool until it passes and stays passing.

Live testing with the test panel

Before you write an eval, talk to the agent yourself. The Test panel holds a live conversation with your draft, one call at a time, exactly as a caller would: how it greets, how it handles a question, whether it reaches for the right tool.

It is the right tool for "does this even work", not for proving a change holds. See Testing before you ship.

What you author

You write each eval in the agent's Tests section: a scenario, its rubric, and the persona the simulated caller runs as. All three, plus line conditions and metrics, are on Writing evals.

The judge

Every scenario is graded by a language-model judge: a separate model that reads the transcript and answers each rubric question. It runs on Claude Sonnet, deliberately stronger than the model the agent itself runs on, which is the standard practice for grading.

It answers each rubric question pass or fail with a one-line reason, and it is told to lean "fail" when genuinely uncertain, because a false pass hides a real defect.

did the agent read the order number back?   pass
did the agent look up the price?            fail
   quoted a price without calling a tool

The three verdicts

• Passed: the agent satisfied the rubric.
• Failed: one or more rubric questions failed. The judge's reason for each tells you what went wrong, and often points straight at the line of the prompt or the tool description at fault.
• Error: the run could not complete (a misconfigured tool, for example).

Structural checks

Alongside the judge, fast model-free structural checks run on the transcript too: catching the agent repeating itself on consecutive turns, stalling after a tool call, or speaking scaffolding it should never voice aloud.

These are the cheap first line that catches obvious regressions before any judge spend.

Running one scenario

Run one scenario on its own for quick feedback while you iterate on a prompt or a tool. The workflow: pick the scenario that covers the behaviour you are changing, run it, read the judge's reasoning, edit, run again. Stay on one scenario until it passes reliably, then run the full batch to check you did not break anything else.

Running a batch

To prove the agent overall, run a batch: the whole suite at once, or a configured sweep. A batch expands across three axes, so you can be deliberate about coverage:

• Scenarios: every eval, or a chosen subset.
• Runs per scenario: repeat each scenario several times. Use five or more for a stability run, because that is what tells you a result holds rather than got lucky once.
• Personas: sweep a scenario across several simulated callers; each scenario runs once per persona.

A batch returns straight away with a batch id and a total run count rather than blocking for a verdict. Open its results to read per-scenario outcomes once it finishes.

Cancel a runaway batch if you launched too large a sweep: queued runs stop, and calls already in flight finish.

Comparing batches

To find out whether one setup beats another, for example a faster speech-to-text model or a reworked prompt, run them as variants in one batch and compare. The comparison groups the runs by variant and reports, for each:

• the success rate;
• the rubric pass-rate, the granular score that separates variants when the all-or-nothing success rate is flat;
• the per-scenario breakdown and per-metric scores;
• per-stage latency for speech-to-text, the language model, and text-to-speech.

The winner is the variant with the highest rubric pass-rate.

What it costs

A run is billed on two legs, whatever the outcome:

• The call itself, charged per minute of audio at the same rate a real call on that agent's model and voice would cost.
• The judge, charged per judge metric the run scores. Each metric carries its own price, so the judge leg is the sum of the metrics active on that scenario. The deterministic structural checks cost nothing.

There is no flat per-run price, because a scenario scoring three judge metrics costs less than one scoring twenty.

The cost is per run, so it multiplies by the expanded total, which is why the run dialog shows an estimated total before you launch. Treat it as an estimate: actual billing follows the metered minutes and metrics.

Where to go next

• Writing evals: scenarios, rubrics, personas, line conditions, metrics.
• Agents: the build-and-publish loop evals gate.
• Writing the prompt: often the right fix when a rubric fails.
• Tools: when a failure is the agent reaching for the wrong tool, or none.
• Calls and analytics: where metrics land on real calls.
• Billing: what runs cost.