Dashboard
Use the dashboard to describe a test, review its starting situation and success rules, run your agent, and understand the outcome. Test files and run history stay in your project.
Published beta.5 opens on Tests. The sidebar groups Tests and Runs under Testing, and Templates, Data and Worlds under Resources. Settings contains workspace and engineering configuration. The interface keeps Sparta's charcoal, orange and soft gray palette, with a collapsible sidebar and a keyboard-accessible menu on narrow screens. Earlier published packages retain their version’s navigation.
Start with the installation guide. For an existing installation, see launching and reopening the dashboard.
Check connector scope for available resources, imports and outcomes before building a test.
For a prepared walkthrough, open Tests → Explore the included example. The six-ticket demo guide covers a fresh package install, review, failed and corrected runs, recorded before/after evidence and repetition. Both scripted agents and all inputs ship with the runtime; no AI provider or additional setup is required.
Explore starter workflows
Templates opens Examples, with Starter workflows above the starting-data templates. Explore starter workflows on both the empty and returning Tests screens leads to the same section. Customer Desk, Team Inbox and Digital Shop show their public missions, supported scope and synthetic-data labels. Browsing creates no worlds.
Choose Open starter to open its existing local experience in a new tab. Inspect records, run the scripted example, connect your agent, introduce interruptions, grade the result and export evidence there. Reopening a starter resumes its active mission and progress while the dashboard host is running. If the browser blocks a tab, use the displayed Open link; retrying a failed launch preserves an already-started attempt. Starter runtimes are separate from the dashboard's main runtime, so discovery and launch still work when that runtime is stopped.
These launches use temporary worlds. Keep the dashboard terminal running; stopping it closes the starters it launched. Make an editable project on each card provides commands for the current installation to create a persistent project or reopen its editor. The starter guide covers custom inputs, agent setup, saved history and portable evidence. Your templates retains reusable saved starting setups and their existing URLs.
Create your first test
On Tests, choose New test and answer What should your agent be able to handle? You can start before your agent or drafting provider is connected. The brief saves privately on the host. Add an optional text policy or starting records, then choose Draft test when AI drafting is configured, or use Build with forms.
An engineer configures Settings → Agent connections and, optionally, Settings → AI drafting once for the workspace. Agent profiles retain the command, working directory, timeout and supported endpoint mappings. Drafting configuration retains an explicit provider, model, private environment-file path and credential variable name; secret values stay in that host file. Engineering links retain the handoff to the same saved draft, including when opened in another tab.
Review the test in three sections:
| Section | Review |
|---|---|
| Starting situation | People, records, relationships, starting time and imported or synthetic data. |
| Work to do | Independent cases, or ordered steps that share one world. |
| Success rules | Expected changes, shared credit allowance and records that must remain untouched. |
Edit the summaries in place. Answer missing account or outcome questions explicitly. Matching names and email addresses never establish a cross-system binding. Unsupported requirements remain visible; excluding one records that it is ungraded. Advanced source editing is available for rules that cannot be represented losslessly by the forms.
Each test uses one grading currency. A refund targeting a payment in another currency needs review and cannot be saved as that test's monetary rule. Use matching-currency starting records or a separate test; the editor preserves the original amounts without converting them.
Choose Save test, or Save and run when a reusable agent connection is selected. That action reviews the current revision and saves ordinary private seed/suite files without creating an exploratory world. New tests start with one repetition and normal conditions. Existing tests retain their other settings; a saved case subset requires an explicit scope choice when reopened here. Ordered steps run together.
Open Run settings in the review to change repetitions, failure conditions, detailed evidence capture or whether worlds remain available after grading. Presets are matched to the workflow's systems using their actual rules; descriptions and target labels explain their scope. Existing custom conditions remain intact with unverified compatibility labeled. These choices save with the same review action and survive reloads. Selecting Normal conditions explicitly removes a saved failure preset. Open saved test files opens the complete task file when a test has already been saved.
If another editor changes a draft, Save edits as a copy preserves your reviewed content and effective run settings together. Recover saved copy resumes that same copy after an interrupted response or reload.
Earlier unfinished setups appear on Tests with a Resume action that returns to their saved progress. Existing files, worlds and earlier setup links remain valid.
Read results and try again
The result starts with its recorded overall outcome and affected cases. Select a finding to compare what should have happened with the observed change. Execution errors, incomplete runs and ungraded cases are distinct; finishing the process is not proof of a passing test. A world-level budget or protection can fail even when one case succeeded.
Open saved rule and private records opens supporting local evidence that may contain sensitive text. The view places the saved rule beside the agent’s captured reply. Captured native support and combined workflows also show the bound account and relevant records in a Recorded before and after table, including refunds, credits, ticket state and subscription changes. Requested changes and protected subscriptions are labeled. Ordered workflows show the start and end of the whole workflow, not an inferred snapshot of each step; shared accounts are identified. The Whole-workflow allowance uses the complete run’s recorded aggregate, including when cases execute independently.
Supporting world changes opens the full recorded comparison, whose scope may include other cases in a shared workflow. Missing observations stay missing; captured excerpts are labeled. Older captures may lack unchanged starting records. A monetary value without a captured denomination stays in minor units rather than borrowing the expected rule’s currency. Identifiers, original findings and raw payloads remain available in details. Settings → AI drafting → Evidence for new tests controls reusable detailed capture, including private starting records. Existing tests keep their capture setting. These recorded comparisons remain unchanged when the retained world advances.
Explore future effects appears when the retained world supports clock controls. Choose the interval, then explicitly open the local records. Worlds creates the baseline and comparison automatically. This advances the retained world; it does not change the original grade.
Each exploration shows its recorded start and end, then refreshes the starting date for the next advance. If another tab changes the clock, the attempt asks you to review the current state instead of saving a misleading comparison. An older connected runtime may need updating before it supports this coordinated operation.
Inspect current world opens the retained world's live records beside the selected finding. Those records can contain sensitive data and later changes; the recorded grade and captured evidence remain separate. Evidence, exports and reproduction contains downloadable evidence, CLI reproduction and execution details.
Run again reuses frozen inputs in fresh worlds without reopening setup or generating another draft. Ordered workflows rerun together. Independent failure subsets remain available in developer reproduction controls. Compare with the previous run keeps the selected case and rule visible in both attempts.
Saved inputs do not freeze agent code or installed dependencies. A selected-case run does not establish that the whole test passes. Changed inputs remain visible in comparison. Reload never reruns or regenerates work; use a displayed recovery action after an uncertain response. Closing the browser leaves execution active while the host remains running.
Earlier world-first creation and reproduction tools
Open Worlds for world tools, Tests → Test files for the advanced suite editor, or the project name in the sidebar for the earlier overview. Existing saved worlds, configurations and deep links continue to work. The following instructions describe that retained workflow.
Create your first test
1. Create a world
On Overview, choose Create a world. Select the systems your agent uses, choose starting data, and give the world a recognizable name.
- Templates provide ready-made starting data you can preview and customize.
- Describe my workflow helps you prepare your own case.
- Import my data starts from your cases or a supported export.
Review the included records, relationships and starting time, then choose Create and explore world. A world is an isolated working copy of that starting data. Creating one does not require an agent or a test suite.
To learn the workflow first, choose Try a guided sample on Overview. The billing, support and combined examples use scripted agents and need no provider key. Compare a failed run with its correction, then return to testing your agent.
2. Inspect the starting state
The world opens on State. Browse its records and relationships to check that your test has the information the agent needs.
Use Failure conditions to add a supported failure, such as a rate limit. Use Clock to advance logical time. Available controls depend on the connector.
Evidence lets you create a mark, a baseline for comparing later changes. A mark does not restore state. Manage → Reset world restores the world's starting state; creating another world leaves this one intact.
3. Connect your agent
Open Connect, then select an agent or choose Create an agent profile. Save its command, working directory and timeout. Follow the connector's instructions to point its SDK at the local world.
Profiles can reference a private environment file and map your agent's endpoint variables. Keep credentials out of version control. Source-account import credentials are separate from the local twin credentials.
Connection checks diagnose endpoint configuration. If your agent waits for a task, use a test to check its normal behavior. Copy developer setup instructions provides a handoff without credentials or answer keys.
4. Define the outcome
Choose Create test in the world. Select the original starting data and clock, or preview an export of its current state. Saving creates a suite: the tasks, starting data and expected outcomes that future runs will use.
In Test suites, edit each task's expected changes and protections. For example, a support task might require a particular status and assignee while protecting unrelated tickets. A billing task might require one specific refund and no changes to another customer.
Combined tests need explicit links between the relevant records. Matching names or email addresses do not establish those links. Written instructions alone are not grading rules; finish the structured expectations or supported YAML rules. Tasks without expectations remain labeled as ungraded.
5. Run fresh tests
Choose Run this suite, select the agent, and review the starting data, outcomes and run settings.
| Mode | When to use it |
|---|---|
| Independent tasks | Test each task in its own fresh world. |
| Shared workflow | Test an ordered set of tasks in the same fresh world. |
Set Repetitions to run the test more than once. Save test keeps the configuration for later; Save and run fresh tests saves it and starts execution. You can save a test before an agent is connected.
Read results and try again
A completed process does not necessarily mean the test passed. Start with Outcome to see the evaluation, then open Evidence for task grades, recorded changes and available reports. A failed protection can fail the overall world even when an individual task was completed correctly.
Open Reproduce to choose the next run:
| Action | What it uses |
|---|---|
| Rerun saved inputs | The captured test data and execution settings. |
| Review current configuration | Current project files and agent settings, reviewed before running. |
| Rerun selected failures from saved inputs | Selected tasks with their saved inputs and protections. |
Saved inputs do not freeze agent code or installed dependencies. A run of selected failures does not establish that the full suite passes. Compare runs identifies compatible tests; when inputs differ, inspect the evidence side by side.
If a submission loses its response, use the displayed recovery action before starting another. Reloading does not rerun work automatically. Keep the launcher terminal open while working; closing the browser leaves running work active.
Custom workflows from your own cases
For the retained import tools, open Worlds → Create a world, then choose Describe my workflow or Import my data. Name the preparation so you can resume it later.
- Add the request and starting situation. Choose an example, project dataset or supported source export for the records the task needs.
- For CSV, JSON or JSONL cases, review the field mapping and select the cases to include. Imported request text does not supply missing business records.
- Review each case's relevant objects, proposed outcomes and protections. Confirm relationships explicitly, especially across connectors.
- Choose Review prepared test files, inspect them, then Save prepared workflow. Continue to Create and explore world and the test flow above.
Preparation is private and resumable. Changed starting data requires reviewing affected outcomes and relationships again.
Account imports and support exports
A scoped Stripe import can collect selected customers, invoices, charges or subscriptions with their supported dependencies. Review the sanitized records, omissions and starting date. The saved data is frozen; reruns do not contact the account again.
Native Zendesk accepts supported offline snapshots. Its Full JSON export path uses extracted ticket envelopes, users and optional organizations. Review the conversation cutoff, starting status, assignment, groups and agent identity. For resolved cases, exclude later resolution messages from the starting context. Live Zendesk ingestion is not supported. See the native workflow reference.
Combined datasets must have compatible starting times and non-overlapping connector data. Imports copy the supported state; they do not reconstruct a complete account history.
Optional model drafting and private data
You can prepare tests manually without a model. Optional OpenAI or Anthropic drafting sends only the content you review and explicitly approve to the selected provider. Supply a private environment-file reference or a session-only key. Review the proposed revision, its assumptions and its outcomes before accepting it.
Accepted private test files live under worlds/private/. Keep environment files, identity maps and source-account credentials out of version control. Default evidence and CI exports exclude private text-bearing inputs; review download previews and prepare de-identified inputs before making a portable CI configuration.
Private reports may omit text or process output. Missing evidence is shown as missing, rather than inferred. Optional advisory analysis remains separate from the test's authoritative verdict.
Return to saved work
Tests shows unfinished drafts and saved tests. Open a draft to continue review, or use Run test to reuse saved inputs and the configured agent. Runs holds previous attempts and comparisons. The project name in the sidebar opens the earlier overview.
In Templates → Your templates, reuse named starting data to create another world. Save as starting setup on a live world previews a reusable export of its current state; this does not preserve its full history or alter the live world.
Runs → Run tools → Saved tests and batches runs several saved configurations in order with one agent profile. Each result remains inspectable, and a failed test does not automatically cancel later tests. Preview the CI export to run reviewed configurations outside the dashboard.
| Destination | Use it for |
|---|---|
| Tests | Describe, review, save and rerun tests; resume unfinished drafts. Test files opens complete task and policy files. |
| Runs | Recorded outcomes, failure investigation, reruns, comparisons, local batches and explicitly imported evidence. |
| Templates | Examples opens starter workflows and starting-data templates; Your templates holds reusable starting setups with optional failure conditions. Success rules belong to tests or the starter's missions. |
| Data | Validated project/library starting records and supported imports. Use records to begin a template draft; no world is created. Private test attachments stay with their drafts. |
| Worlds | Live and retained world instances, records, requests, failure conditions, clock, evidence, connections and lifecycle controls. |
| Settings → Workspace | Local project/runtime tools, supported data sources, delivery testing and CI workflow export. No personal account is required. |
| Settings → Agent connections | Reusable local agent commands, endpoint mappings and private environment-file references. |
| Settings → AI drafting | Provider/model configuration, private credential references and evidence defaults. Secret values stay on the host; runtime and per-world credentials retain their existing controls. |
CI workflow export prepares GitHub Actions files; PR status and history remain in GitHub. Delivery testing exposes the existing scoped browser adapters, including its registered MCP host option. These tools live within their supported workflows, not separate integration, PR-history or general MCP products. Templates uses existing starting setups; failure-injection rules remain contextual controls. Earlier section URLs remain supported, including the superseded preview labels. Selecting a destination never generates a test, starts a run or provisions a world.