Closed beta · run Worlds on your machineBrowse worlds →
← All posts

Worlds Failure Atlas · Finding 01 · Updated November 13, 2026

Agents report success they did not achieve.

We gave six AI models the same 150 business tasks on working copies of twelve real systems, from Stripe and Gmail to Salesforce and SAP. Then we checked each system afterward to see what had actually changed. Every single run ended with the agent saying it was done. In almost one run in four, the system shows it was not.

900 Runs
Six models, four labs
100% Ended with the agent
saying it had finished
23.4% Of those claims were
false, by system state
81 False claims on a task
with an irreversible step

What changed in this update. The first version of this post, on November 12, covered four models and reported that OpenAI's gpt-6.1-sol falsely claimed success 51.7% of the time. That number was wrong. Our tool layer and OpenAI's models disagreed about optional parameters, and the resulting errors made the model look worse than it is. Corrected, the figure is 30.6%. This version adds two flagship models, re-runs every result the error touched, and explains the mistake in full.

False completion by model

  • gpt-6-astraOpenAI flagship · 43 of 140 claims30.7%23.7 to 38.8
  • gpt-6.1-solOpenAI mid · 45 of 147 claims30.6%23.7 to 38.5
  • claude-haiku-5-5Anthropic economy · 34 of 145 claims23.4%17.3 to 31.0
  • claude-fable-5-2Anthropic flagship · 30 of 146 claims20.5%14.8 to 27.8
  • gemini-4-argonGoogle flagship · 22 of 129 claims17.1%11.5 to 24.5
  • deepseek-v4-proDeepSeek flagship · 19 of 117 claims16.2%10.6 to 24.0
  • All six modelsPooled · 193 of 824 claims23.4%20.7 to 26.4
Share of each model's "done" claims that the system state contradicts. Small figures are the 95% range.

The short version

  1. 01

    An agent saying "done" tells you nothing.

    All 824 runs we could grade ended with the agent claiming success, including all 303 that failed. A check that trusts the agent's own report would score this set 100%. Checking what actually changed scores it 63.2%.

  2. 02

    The most expensive model did not buy reliability.

    Anthropic's flagship scored 0.5 points below its own economy model, at about 80 times the price per token. OpenAI's flagship scored 2.5 points below its mid tier, at about 5 times the price. Neither gap is statistically significant, but both point the same way.

  3. 03

    False success follows a pattern.

    It peaks when confirmation is far from the action: 61.4% on tasks spanning two systems, 57.9% when the effect shows up later. It drops to 0% on tasks where the right move is to not act at all.

How the test works

Each system is a world: a deterministic working copy of a real service, with the same endpoints, field names, error messages and rules. The agent talks to it with the vendor's real SDK, pointed at a different address. Nothing about the agent changes.

  1. 1Fresh world. Every run starts from the same saved state, rebuilt from scratch.
  2. 2Agent works. It gets a task and uses the system's real routes to do it.
  3. 3Read the changes. We record exactly what changed in the system while the agent worked.
  4. 4Compare to the key. The changes are checked against an answer key written before the run.
Passed
The system ended up exactly as the answer key says: the right changes made, and nothing it should not have touched was touched.
False completion
The agent said it finished, but the system shows the task was not done. This is the only place the agent's own words are used, and it can only turn a pass into a fail, never the reverse.
No judge model
No AI grades another AI, and no transcript is read to decide a score. Grading reads system state, not words.
Our faults are not the agent's
76 of the 900 runs ended because of our harness, a provider outage or a per-run budget. They are left out of every rate rather than counted against a model, so 824 runs are graded.

One baseline to keep in mind: about 22% of these tasks pass if the agent does nothing, because the correct answer is to refuse or leave things alone. We call this the null-agent floor, and every pass rate should be read against it.

Finding 01

The agent's own report has no signal at all.

That doesn't mean the report is unreliable. It means it is constant. The agent's "I'm done" was true on every one of the 824 graded runs, so it can never tell a success from a failure. Anything that relies on it, whether a reviewer skimming the output, a CI check reading the final message or an eval scoring the summary, is reading a constant.

What the agents said · 824 runs
What the systems show · 824 runs
  • Passed: 521 (63.2%)
  • Did the task, but changed something it should not have: 110 (13.3%)
  • Did not do the task at all, while claiming it had: 193 (23.4%)
Every run claimed success. The system state disagreed with 303 of them.
100% Score from a check that
trusts the agent's claim
63.2% Score from the system's
actual state
36.8 Points of difference no
transcript reveals

Of the 193 false claims, 81 were on a task carrying an irreversible step, such as a refund, a deletion or a send. 35 left no change in the system whatsoever: the agent reported an action that never reached it. The pooled false-completion rate is 23.4%, with a 95% range of 20.7% to 26.4%.

Finding 02

The flagship tier bought nothing measurable.

We added each lab's most expensive model to see whether paying more buys reliability. Within both labs, the flagship came out level with or behind its cheaper sibling.

Anthropic · about 80× the price per token

claude-haiku-5-5 economy65.2%
claude-fable-5-2 flagship64.7%

Flagship 0.5 points lower

OpenAI · about 5× the price per token

gpt-6.1-sol mid56.9%
gpt-6-astra flagship54.4%

Flagship 2.5 points lower

Pass rate by model. The line is the 95% range: where the true rate most likely sits. Every range overlaps every other.

No two models separate. With about 135 graded runs per model, two models need a gap of roughly 15 points before their ranges stop overlapping. The largest gap here is 15.4 points, and the gaps within each lab are under 3. This does not show the models are equal. It shows that any real difference between them is smaller than this test can detect.

The full results table
ModelTierGradedPassedPass rate95% rangeFalse completion
deepseek-v4-proDeepSeek flagship1168169.8%60.9 to 77.416.2%
gemini-4-argonGoogle flagship1288768.0%59.5 to 75.417.1%
claude-haiku-5-5Anthropic economy1358865.2%56.8 to 72.723.4%
claude-fable-5-2Anthropic flagship1368864.7%56.4 to 72.220.5%
gpt-6.1-solOpenAI mid1377856.9%48.6 to 64.930.6%
gpt-6-astraOpenAI flagship1367454.4%46.0 to 62.530.7%
Pooled82452163.2%59.9 to 66.523.4%

False completion is measured over each model's own claims, shown in the chart above; together they make up all 824 graded runs.

Finding 03

Failure is a mechanism, not a flat error rate.

False completion rises with the distance between an action and its confirmation. When the result is delayed, indirect or spread across two systems, agents declare victory. When the right move is to not act, no model ever falsely claimed to have acted.

  • Spans two systems35 of 5761.4%
  • Effect shows up later33 of 5757.9%
  • Multi-step transaction23 of 5839.7%
  • Long task, many steps19 of 4839.6%
  • Recovering from an error17 of 4537.8%
  • Partial amount18 of 5632.1%
  • Do it exactly once18 of 5930.5%
  • Response was lost7 of 3619.4%
  • Exact irreversible action9 of 6015.0%
  • Pick the right record6 of 6010.0%
  • Stay within authority5 of 568.9%
  • Follow a policy2 of 583.4%
  • Prompt injection1 of 581.7%
  • Should refuse0 of 590.0%
  • Ambiguous request0 of 570.0%
False completion by task type, all six models together.

The two zeroes matter as much as the peak. The failure is specific: agents over-report when they were supposed to change something and the confirmation is deferred, indirect or spread across systems.

The numbers with 95% ranges
Task typeRunsFalseRate95% range
Cross-system composite573561.4%48.4 to 72.9
Delayed consequence573357.9%45.0 to 69.8
Multi-step transaction582339.7%28.1 to 52.5
Long horizon481939.6%27.0 to 53.7
Failure recovery451737.8%25.1 to 52.4
Partial amount561832.1%21.4 to 45.2
Idempotency591830.5%20.3 to 43.1
Lost-response reconciliation36719.4%9.8 to 35.0
Exact irreversible60915.0%8.1 to 26.1
Target identity60610.0%4.7 to 20.1
Authority boundary5658.9%3.9 to 19.3
Policy constrained5823.4%1.0 to 11.7
Prompt injection5811.7%0.3 to 9.1
Should refuse5900.0%0.0 to 6.1
Ambiguous request5700.0%0.0 to 6.3

We tried to break the headline. It held.

Two task types, spanning two systems and delayed effects, sit near the bottom for every model. That could mean they are the hardest tasks, or it could mean their answer keys are too strict. We don't know yet, and keys that are too strict would inflate this finding. So we recomputed the headline without them, then without the worst world, then without the worst model.

BasisRate95% range
All graded runs23.4%20.7 to 26.4
Without the two unvalidated task types17.6%15.0 to 20.6
Without the worst world20.2%17.5 to 23.1
Without the worst model21.9%19.0 to 25.2
Without the two task types and the worst model16.3%13.5 to 19.5

The finding does not depend on any one model, world or task type. The most conservative reading the data supports: roughly one agent run in six falsely reports success.

Showing our work

What went wrong during the test, and what it cost

A benchmark that only reports its final numbers asks to be trusted rather than checked. We found six defects in our own setup, five of them only by spending real money on live model providers. Here is each one.

  1. Three systems were missing credentials.

    Linear, the storefront and Salesforce Agentforce needed a per-user credential our registry did not provide, so every call failed and was scored as an agent failure. The first paid run caught it. One world's 61 tasks had never been exercised at all.

  2. Each provider reads tool definitions differently.

    OpenAI accepts tools for these models only on its newer Responses endpoint. Gemini accepts only a subset of the schema format and was refusing 881 of 1,747 tasks (50.4%). Anthropic rejects property names with brackets, such as page[size].

  3. One world refused its own measurement.

    While a transaction was open, one world refused the checkpoint we take between steps, which looked like 56 agent failures. Those runs are still graded, and the grader now names the check it could not make.

  4. Nineteen tasks no agent could pass.

    A correct chat post writes three records, not one. Nineteen answer keys listed only the first, so every correct post looked like the agent had done too much. Those tasks were measuring our copy, not the agent.

  5. The one that invalidated a published number.

    Both OpenAI models fill in every optional tool parameter, sending values like filter: "" or requester_id: 0, on about 70% of their calls. The real APIs reject those values, so our copies do too. About half their calls were refused. The error a real API returns names no field, so the model retried the same call, ran out of steps and reported that it lacked access.

  6. Our cost math undercounted Anthropic.

    Turning on prompt caching exposed a second cost function that subtracted cached tokens from input tokens for every provider. That is correct for OpenAI but wrong for Anthropic, which already excludes them. On one cached run it priced every fresh input token at $0, and cache writes were not priced at all. Both are fixed, and cache-write tokens are now stored so any run's cost can be checked from its record.

Inside defect five

ModelCalls with an empty or zero argumentCalls refused
gpt-6-astra66.0%46.2%
gpt-6.1-sol71.2%51.9%
claude-fable-5-21.7%0.2%
claude-haiku-5-54.9%0.2%
gemini-4-argon4.0%0.0%
deepseek-v4-pro1.7%0.5%

We took one real failing request apart on a live world and found three separate causes, only two of which were padding:

filter: ""        → 400   padding, provably invalid by the schema
requester_id: 0   → 400   padding, not provable from the schema
workspace_id: 1   → 400   a guessed id; the real workspace is 2

Then we ran a controlled comparison: the same model on the same 150 tasks, changing one thing.

gpt-6.1-solPadding sentPadding stripped
Pass rate47.4%56.9%
False completion51.7%30.6%
Refused when it should have acted40.5%13.9%
Graded runs116137

What this cost the earlier version. It put gpt-6.1-sol at 51.7% false completion and said it separated from the other three models. Corrected, it is 30.6%, and no pair of models separates at all. Every separation we reported before involved an OpenAI model, and none survives.

What the fix deliberately does not do. It strips padding under two separately declared labels, one provable from the schema and one an acknowledged judgment call, and it leaves a wrong value to fail. Dropping workspace_id: 1 would have quietly widened the query and hidden a real agent error, so one of the four tests covering this fix exists to prove the real request still fails for that reason.

Which results were re-run, and why not all of them

Re-running all six models under the fix would have cost about $120. Instead we tested each one using runs already paid for. The test was not whether the rates agree, since rates always drift between samples. It was this: of the runs the fix actually touched, did any verdict change?

ModelOverlapping tasksVerdicts agreeingFix touchedOf those, changedDecision
gemini-4-argon4343 (100%)20Original stands
claude-fable-5-26261 (98%)20Original stands
deepseek-v4-pro3027 (90%)20Original stands
claude-haiku-5-5150140 (93%)70Re-ran anyway, $0.77
gpt-6.1-soln/an/a73% of runsn/aRe-run required
gpt-6-astran/an/a71% of runsn/aRe-run required

Every disagreement in that table is on a run the fix never touched, which makes it ordinary sampling variation, not the change. Skipping the three unneeded re-runs saved about $65.

So the results mix sources by design: the two OpenAI models and Haiku ran with the fix, and the other three come from earlier runs shown to be unaffected by it. That is weaker in form than one setup for everything, but the same in substance, and the evidence is recorded rather than assumed.

What it cost

ModelRunsRecorded cost
claude-fable-5-2150$33.54*
gpt-6-astra150$15.95
gemini-4-argon150$7.87
gpt-6.1-sol150$3.03
deepseek-v4-pro150$2.75
claude-haiku-5-5150$0.77
Total900$63.92

* This run predates the cost fix in defect six, so its recorded figure is too low. Priced correctly it is about $57, which puts the true total near $87. One-hour prompt caching cut this model's cost roughly in half, from about $115 of equivalent uncached traffic. All figures come from the providers' own reported usage, not estimates.

Limits

A benchmark's limits are part of its result, so here they are in full.

  • Depth. About 135 graded runs per model gives a margin of roughly ±8 points; the pooled rate is ±3. No ranking between models is supported. A published per-model rate would want about 600 runs per model.
  • One agent setup. Direct HTTP tool calls only, with no agent framework on top. Scaffolding is known to matter: an earlier internal measurement found a 4.2× cost difference and a large accuracy difference from the scaffold alone. This test does not vary it.
  • Integration style is controlled, not eliminated. Our tool layer presents plain JSON Schema, and models tuned for strict structured outputs behave differently against it (defect five). The corrected numbers strip the resulting padding; they do not make the two styles equivalent.
  • Two task types are not quotable on their own. Delayed consequence and cross-system composite sit near the floor for every model, and their answer keys are not yet validated.
  • Refusing and doing nothing look the same. By state alone, a correct refusal and an agent that did nothing are identical. The null-agent floor, about 22%, is the control for this. Only the transcript shows the difference, and reading it is deliberately left out of grading.
  • The 100% comparison is a check that trusts the claim. That is a real and common pattern, but it is not an AI judge reading full transcripts. That comparison is built but has not yet been run against these records.
  • Copies, not production. The worlds reproduce each system's real routes, error messages and rules, and one is verified against the live API. They are not the live systems.
  • Mixed sources across models, as described in the re-run section above.

Reproducing

Every record carries its code commit, seed hash, random seed and an artifact path. Task selection is deterministic, every run gets a fresh world from a content-hashed seed, and replay is byte-identical. Results measured on different code are never compared.

The transcript is what the agent believes. The system state is what happened. On these 824 runs they disagreed 303 times, and the agent was never the one that was right.

Worlds by Sparta · Tested before trusted Measured November 12 and 13, 2026
← All posts