Transcript grading vs outcome grading

Transcript grading evaluates what the agent said or recorded in its trace. Outcome grading checks the state produced by its actions. The two answer different questions and can be used together.

Inspect the system state

A world records supported mutations as an agent acts. A mark gives the test a comparison baseline; the diff describes records created, updated or deleted after it. An expectation can check both the requested action and protected records that should remain unchanged.

A plausible reply does not prove that the requested record changed. Equally, a correct state change does not establish that the reply was clear or appropriate. Keep those judgments distinct.

Understand the result's limits

A grade depends on the task's expectations, the starting data, the supported API behavior and the evidence available. A task without an authoritative expected outcome may remain ungraded. Optional model judgments are advisory and may involve sending selected content to a model provider.

The Worlds beta includes the implemented Stripe subset and native Zendesk, plus supported combined workflows. Stripe has dated live-API probe evidence; native Zendesk live API qualification is pending. The known limitations apply to every report, including a passing one.

Repeat the case deliberately

Use the same saved inputs to compare an agent correction. Deterministic world replay means the same request sequence has the same world result; a model may choose another sequence. Compare compatible runs, inspect extra changes, and retain enough reviewed evidence to explain the outcome.

Read the determinism reference, or join the beta for the local dashboard and CLI guides. Synthetic samples illustrate this workflow without making a claim about customer or model performance.