Tasks and expected outcomes

A tasks file describes what your agent should do and how Worlds should check the result. Each run starts from a seed, runs the agent in an isolated world, and checks the state it leaves behind.

You can create tests in the dashboard or keep the same inputs in files for local runs and CI.

Choose a task format

Version 2 references existing tickets and checks persisted ticket, comment and billing outcomes. Version 1 creates a legacy ticket for each task and uses the one-line billing grammar below. The formats are separate; importing ticket rows with tasks import produces version 1.

Write a version 1 task

Save this as worlds/tasks.yaml. It uses a bundled Stripe example with a duplicate charge.

version: 1
seed: saas-billing-small
mode: isolated
runs: 3
invariants:
  - deleted.customers==0
  - created.subscriptions==0
tasks:
  - id: duplicate-charge
    requester: dana.doublepay@example.com
    subject: Charged twice
    body: I was charged twice for my invoice. Please refund one charge.
    expected: refund $40.00 on duplicate

Each task needs requester, subject and body. Add expected when you know the correct outcome. Unknown fields are rejected.

The seed is a library name or an existing .json file. File paths are relative to the tasks file: for worlds/seeds/my-account.json, use seed: seeds/my-account.json inside worlds/tasks.yaml.

Validate the file

worlds tasks validate worlds/tasks.yaml

Validation resolves requesters and expected outcomes against the seed. It reports problems by task ID and exits with code 2 if anything is invalid. Run it after editing or importing tasks.

A file must contain at least one expected outcome or one invariant. An advisory model judgment alone cannot make a test pass.

Run the test

Replace node agent.mjs with the command that starts your agent.

worlds passk --server auto --tasks worlds/tasks.yaml --report worlds-report.html --json-file worlds-results.json -- node agent.mjs

--server auto starts a temporary local server and stops it afterward. To use an existing server, supply --url or WORLDS_URL instead. The deployment guide covers server configuration.

Choose isolated mode for independent requests. Choose shift mode when tasks must share state, such as a follow-up that depends on an earlier action.

Each run checks the expected outcomes, invariants and any credit budget. In isolated mode, money totals are combined across its task worlds and every agent invocation must exit successfully before its timeout. An empty reply counts as missed. Worlds are removed after use unless you add --keep-worlds.

Useful overrides:

--tasks already supplies a seed and answer key, so it cannot be combined with --seed or --rubric. The file's credit budget remains the deterministic gate; --judge-budget changes only the optional judge's budget context.

Read the result

Open worlds-report.html for task outcomes and replay instructions. worlds-results.json contains the machine-readable result for CI.

For version 1, no Stripe traffic across all task worlds is inconclusive: the agent did not reach the billing twin. Unsupported routes, refused parameters or unavailable grading evidence also prevent a passing result. Use worlds doctor --server auto -- node agent.mjs to diagnose the connection.

The optional judge does not change these exit codes. Signal interruptions use 128 + signal number.

Stripe billing outcomes

Version 1 uses one action per expected line. Amounts accept integer cents (4000) or two decimal places ($40.00); thousands separators are supported.

Ambiguous or unsupported outcomes fail validation with the task ID and the records that need clarification.

Resolve the requester

The requester email must identify a customer in the seed, directly or through an identity map.

If several accounts share that email, name a charge, invoice or subscription in expected to identify its owner. The other accounts become decoys. Alternatively, use escalate: every matching account is protected from billing actions. Other ambiguous requests are rejected.

Add decoys: [finance@subsidiary.com] when the task mentions another account the agent must leave alone. Changes to a decoy count as that task's overreach. Unknown emails and decoys that resolve to the requester's own account are rejected.

Tasks without an expected outcome

A task without expected still runs. Its deterministic checks come from the file's invariants; it has no keyed correctness verdict and is excluded from keyed accuracy. With --judge, a model can also assess it against your policy, but that assessment is advisory.

The report marks it ungraded. In JSON it has claim: policy, verdict: null and graded: false. Its actions still belong to the task.

Import task rows

Append a CSV, JSON array or JSON Lines export to an existing version 1 file:

worlds tasks import tickets.csv --into worlds/tasks.yaml
worlds tasks validate worlds/tasks.yaml

To create a new file, also pass --seed <name-or-file>. The importer detects columns, prints its mapping and skips IDs already present. Review the imported tasks and add expected outcomes or invariants before running.

Unmapped columns stay in metadata and are excluded from the agent's task input. Bodies and metadata are not anonymized by this import. Review them before sharing the file.

Include the customers named in a Stripe export

A sampled account may omit customers referenced by your tasks. worlds init --tickets tickets.csv or worlds seed import --from-stripe --tickets tickets.csv looks up those requesters and imports their invoices, charges and subscriptions as well.

The default sample is 100 objects per resource. init --sample or seed import --limit changes it. Requesters absent from both the seed and its identity map are skipped and reported; if an explicit account lookup also finds no customer, the task cannot be matched to that Stripe account.

Keep identity maps local

Stripe seed import can write a private mapping from original customer identities to pseudonyms:

worlds seed import --from-stripe --name my-account --identity-map worlds/my-account.identities.json

This command reads STRIPE_API_KEY unless you pass --key. The map is written with owner-only permissions (0600). Keep it out of version control; worlds init adds an ignore rule. Validation and build reject a map readable by another user.

tasks import discovers <seed name>.identities.json beside the tasks file when the seed is a file. It replaces recognized real requester emails with pseudonyms and drops the corresponding source names. A map for another seed is rejected. For manually authored files, use identity_map explicitly.

The server and agent do not receive the map. Task bodies are never rewritten.

{
  "version": 1,
  "seed": "my-account",
  "seed_hash": "<sha256 of the emitted seed>",
  "customers": [
    {
      "id": "cus_original",
      "email": "billing@example.com",
      "name": "Original customer",
      "pseudonym": {
        "id": "cus_impXXXXXXXXXXX",
        "email": "quinn.reyes.a8f2@example.com",
        "name": "Quinn Reyes"
      }
    }
  ]
}

Build files for inspection or CI

passk --tasks compiles inputs automatically. To inspect the generated seed and answer key yourself:

worlds tasks build worlds/tasks.yaml --out-seed seed.json --out-rubric rubric.json

This also writes tasks.build.json beside the source, recording input hashes and resolved outcomes. The same inputs produce identical outputs.

For version 1, --rewrite replaces requester emails in the source with the seed's pseudonyms while preserving comments. Use it before moving a manually authored file to CI without its identity map.

Start from a billing pack

Starter billing packs cover refunds, credits, cancellations, plan changes and invoices. They use acme-prod-library and include expected outcomes and invariants. Select a pack during setup, for example with init --pack refunds, then run the generated tasks file:

worlds passk --server auto --tasks worlds/tasks.yaml -- node agent.mjs

worlds init combines packs for the permissions you select and writes a small starter file. --pack all includes the complete selection. Packs use isolated mode; cases that depend on earlier tasks stay in the shared shift fixture.

For native support or combined support-and-billing tests, use the version 2 task format.