Worlds Failure Atlas · Finding 01 · Updated November 13, 2026
Agents report success they did not achieve.
We gave six AI models the same 150 business tasks on working copies of twelve real systems, from Stripe and Gmail to Salesforce and SAP. Then we checked each system afterward to see what had actually changed. Every single run ended with the agent saying it was done. In almost one run in four, the system shows it was not.
Six models, four labs
saying it had finished
false, by system state
with an irreversible step
What changed in this update. The first version of this post, on November 12, covered four models and reported that OpenAI's gpt-6.1-sol falsely claimed success 51.7% of the time. That number was wrong. Our tool layer and OpenAI's models disagreed about optional parameters, and the resulting errors made the model look worse than it is. Corrected, the figure is 30.6%. This version adds two flagship models, re-runs every result the error touched, and explains the mistake in full.
False completion by model
The short version
-
01
An agent saying "done" tells you nothing.
All 824 runs we could grade ended with the agent claiming success, including all 303 that failed. A check that trusts the agent's own report would score this set 100%. Checking what actually changed scores it 63.2%.
-
02
The most expensive model did not buy reliability.
Anthropic's flagship scored 0.5 points below its own economy model, at about 80 times the price per token. OpenAI's flagship scored 2.5 points below its mid tier, at about 5 times the price. Neither gap is statistically significant, but both point the same way.
-
03
False success follows a pattern.
It peaks when confirmation is far from the action: 61.4% on tasks spanning two systems, 57.9% when the effect shows up later. It drops to 0% on tasks where the right move is to not act at all.
How the test works
Each system is a world: a deterministic working copy of a real service, with the same endpoints, field names, error messages and rules. The agent talks to it with the vendor's real SDK, pointed at a different address. Nothing about the agent changes.
- 1Fresh world. Every run starts from the same saved state, rebuilt from scratch.
- 2Agent works. It gets a task and uses the system's real routes to do it.
- 3Read the changes. We record exactly what changed in the system while the agent worked.
- 4Compare to the key. The changes are checked against an answer key written before the run.
- Passed
- The system ended up exactly as the answer key says: the right changes made, and nothing it should not have touched was touched.
- False completion
- The agent said it finished, but the system shows the task was not done. This is the only place the agent's own words are used, and it can only turn a pass into a fail, never the reverse.
- No judge model
- No AI grades another AI, and no transcript is read to decide a score. Grading reads system state, not words.
- Our faults are not the agent's
- 76 of the 900 runs ended because of our harness, a provider outage or a per-run budget. They are left out of every rate rather than counted against a model, so 824 runs are graded.
One baseline to keep in mind: about 22% of these tasks pass if the agent does nothing, because the correct answer is to refuse or leave things alone. We call this the null-agent floor, and every pass rate should be read against it.
Finding 01
The agent's own report has no signal at all.
That doesn't mean the report is unreliable. It means it is constant. The agent's "I'm done" was true on every one of the 824 graded runs, so it can never tell a success from a failure. Anything that relies on it, whether a reviewer skimming the output, a CI check reading the final message or an eval scoring the summary, is reading a constant.
- Passed: 521 (63.2%)
- Did the task, but changed something it should not have: 110 (13.3%)
- Did not do the task at all, while claiming it had: 193 (23.4%)
trusts the agent's claim
actual state
transcript reveals
Of the 193 false claims, 81 were on a task carrying an irreversible step, such as a refund, a deletion or a send. 35 left no change in the system whatsoever: the agent reported an action that never reached it. The pooled false-completion rate is 23.4%, with a 95% range of 20.7% to 26.4%.
Finding 02
The flagship tier bought nothing measurable.
We added each lab's most expensive model to see whether paying more buys reliability. Within both labs, the flagship came out level with or behind its cheaper sibling.
Anthropic · about 80× the price per token
Flagship 0.5 points lower
OpenAI · about 5× the price per token
Flagship 2.5 points lower
No two models separate. With about 135 graded runs per model, two models need a gap of roughly 15 points before their ranges stop overlapping. The largest gap here is 15.4 points, and the gaps within each lab are under 3. This does not show the models are equal. It shows that any real difference between them is smaller than this test can detect.
The full results table
| Model | Tier | Graded | Passed | Pass rate | 95% range | False completion |
|---|---|---|---|---|---|---|
| deepseek-v4-pro | DeepSeek flagship | 116 | 81 | 69.8% | 60.9 to 77.4 | 16.2% |
| gemini-4-argon | Google flagship | 128 | 87 | 68.0% | 59.5 to 75.4 | 17.1% |
| claude-haiku-5-5 | Anthropic economy | 135 | 88 | 65.2% | 56.8 to 72.7 | 23.4% |
| claude-fable-5-2 | Anthropic flagship | 136 | 88 | 64.7% | 56.4 to 72.2 | 20.5% |
| gpt-6.1-sol | OpenAI mid | 137 | 78 | 56.9% | 48.6 to 64.9 | 30.6% |
| gpt-6-astra | OpenAI flagship | 136 | 74 | 54.4% | 46.0 to 62.5 | 30.7% |
| Pooled | 824 | 521 | 63.2% | 59.9 to 66.5 | 23.4% |
False completion is measured over each model's own claims, shown in the chart above; together they make up all 824 graded runs.
Finding 03
Failure is a mechanism, not a flat error rate.
False completion rises with the distance between an action and its confirmation. When the result is delayed, indirect or spread across two systems, agents declare victory. When the right move is to not act, no model ever falsely claimed to have acted.
The two zeroes matter as much as the peak. The failure is specific: agents over-report when they were supposed to change something and the confirmation is deferred, indirect or spread across systems.
The numbers with 95% ranges
| Task type | Runs | False | Rate | 95% range |
|---|---|---|---|---|
| Cross-system composite | 57 | 35 | 61.4% | 48.4 to 72.9 |
| Delayed consequence | 57 | 33 | 57.9% | 45.0 to 69.8 |
| Multi-step transaction | 58 | 23 | 39.7% | 28.1 to 52.5 |
| Long horizon | 48 | 19 | 39.6% | 27.0 to 53.7 |
| Failure recovery | 45 | 17 | 37.8% | 25.1 to 52.4 |
| Partial amount | 56 | 18 | 32.1% | 21.4 to 45.2 |
| Idempotency | 59 | 18 | 30.5% | 20.3 to 43.1 |
| Lost-response reconciliation | 36 | 7 | 19.4% | 9.8 to 35.0 |
| Exact irreversible | 60 | 9 | 15.0% | 8.1 to 26.1 |
| Target identity | 60 | 6 | 10.0% | 4.7 to 20.1 |
| Authority boundary | 56 | 5 | 8.9% | 3.9 to 19.3 |
| Policy constrained | 58 | 2 | 3.4% | 1.0 to 11.7 |
| Prompt injection | 58 | 1 | 1.7% | 0.3 to 9.1 |
| Should refuse | 59 | 0 | 0.0% | 0.0 to 6.1 |
| Ambiguous request | 57 | 0 | 0.0% | 0.0 to 6.3 |
We tried to break the headline. It held.
Two task types, spanning two systems and delayed effects, sit near the bottom for every model. That could mean they are the hardest tasks, or it could mean their answer keys are too strict. We don't know yet, and keys that are too strict would inflate this finding. So we recomputed the headline without them, then without the worst world, then without the worst model.
| Basis | Rate | 95% range |
|---|---|---|
| All graded runs | 23.4% | 20.7 to 26.4 |
| Without the two unvalidated task types | 17.6% | 15.0 to 20.6 |
| Without the worst world | 20.2% | 17.5 to 23.1 |
| Without the worst model | 21.9% | 19.0 to 25.2 |
| Without the two task types and the worst model | 16.3% | 13.5 to 19.5 |
The finding does not depend on any one model, world or task type. The most conservative reading the data supports: roughly one agent run in six falsely reports success.
Showing our work
What went wrong during the test, and what it cost
A benchmark that only reports its final numbers asks to be trusted rather than checked. We found six defects in our own setup, five of them only by spending real money on live model providers. Here is each one.
-
Three systems were missing credentials.
Linear, the storefront and Salesforce Agentforce needed a per-user credential our registry did not provide, so every call failed and was scored as an agent failure. The first paid run caught it. One world's 61 tasks had never been exercised at all.
-
Each provider reads tool definitions differently.
OpenAI accepts tools for these models only on its newer Responses endpoint. Gemini accepts only a subset of the schema format and was refusing 881 of 1,747 tasks (50.4%). Anthropic rejects property names with brackets, such as
page[size]. -
One world refused its own measurement.
While a transaction was open, one world refused the checkpoint we take between steps, which looked like 56 agent failures. Those runs are still graded, and the grader now names the check it could not make.
-
Nineteen tasks no agent could pass.
A correct chat post writes three records, not one. Nineteen answer keys listed only the first, so every correct post looked like the agent had done too much. Those tasks were measuring our copy, not the agent.
-
The one that invalidated a published number.
Both OpenAI models fill in every optional tool parameter, sending values like
filter: ""orrequester_id: 0, on about 70% of their calls. The real APIs reject those values, so our copies do too. About half their calls were refused. The error a real API returns names no field, so the model retried the same call, ran out of steps and reported that it lacked access. -
Our cost math undercounted Anthropic.
Turning on prompt caching exposed a second cost function that subtracted cached tokens from input tokens for every provider. That is correct for OpenAI but wrong for Anthropic, which already excludes them. On one cached run it priced every fresh input token at $0, and cache writes were not priced at all. Both are fixed, and cache-write tokens are now stored so any run's cost can be checked from its record.
Inside defect five
| Model | Calls with an empty or zero argument | Calls refused |
|---|---|---|
| gpt-6-astra | 66.0% | 46.2% |
| gpt-6.1-sol | 71.2% | 51.9% |
| claude-fable-5-2 | 1.7% | 0.2% |
| claude-haiku-5-5 | 4.9% | 0.2% |
| gemini-4-argon | 4.0% | 0.0% |
| deepseek-v4-pro | 1.7% | 0.5% |
We took one real failing request apart on a live world and found three separate causes, only two of which were padding:
filter: "" → 400 padding, provably invalid by the schema
requester_id: 0 → 400 padding, not provable from the schema
workspace_id: 1 → 400 a guessed id; the real workspace is 2
Then we ran a controlled comparison: the same model on the same 150 tasks, changing one thing.
| gpt-6.1-sol | Padding sent | Padding stripped |
|---|---|---|
| Pass rate | 47.4% | 56.9% |
| False completion | 51.7% | 30.6% |
| Refused when it should have acted | 40.5% | 13.9% |
| Graded runs | 116 | 137 |
What this cost the earlier version. It put gpt-6.1-sol at 51.7% false completion and said it separated from the other three models. Corrected, it is 30.6%, and no pair of models separates at all. Every separation we reported before involved an OpenAI model, and none survives.
What the fix deliberately does not do. It strips padding under two separately declared labels, one provable from the schema and one an acknowledged judgment call, and it leaves a wrong value to fail. Dropping workspace_id: 1 would have quietly widened the query and hidden a real agent error, so one of the four tests covering this fix exists to prove the real request still fails for that reason.
Which results were re-run, and why not all of them
Re-running all six models under the fix would have cost about $120. Instead we tested each one using runs already paid for. The test was not whether the rates agree, since rates always drift between samples. It was this: of the runs the fix actually touched, did any verdict change?
| Model | Overlapping tasks | Verdicts agreeing | Fix touched | Of those, changed | Decision |
|---|---|---|---|---|---|
| gemini-4-argon | 43 | 43 (100%) | 2 | 0 | Original stands |
| claude-fable-5-2 | 62 | 61 (98%) | 2 | 0 | Original stands |
| deepseek-v4-pro | 30 | 27 (90%) | 2 | 0 | Original stands |
| claude-haiku-5-5 | 150 | 140 (93%) | 7 | 0 | Re-ran anyway, $0.77 |
| gpt-6.1-sol | n/a | n/a | 73% of runs | n/a | Re-run required |
| gpt-6-astra | n/a | n/a | 71% of runs | n/a | Re-run required |
Every disagreement in that table is on a run the fix never touched, which makes it ordinary sampling variation, not the change. Skipping the three unneeded re-runs saved about $65.
So the results mix sources by design: the two OpenAI models and Haiku ran with the fix, and the other three come from earlier runs shown to be unaffected by it. That is weaker in form than one setup for everything, but the same in substance, and the evidence is recorded rather than assumed.
What it cost
| Model | Runs | Recorded cost |
|---|---|---|
| claude-fable-5-2 | 150 | $33.54* |
| gpt-6-astra | 150 | $15.95 |
| gemini-4-argon | 150 | $7.87 |
| gpt-6.1-sol | 150 | $3.03 |
| deepseek-v4-pro | 150 | $2.75 |
| claude-haiku-5-5 | 150 | $0.77 |
| Total | 900 | $63.92 |
* This run predates the cost fix in defect six, so its recorded figure is too low. Priced correctly it is about $57, which puts the true total near $87. One-hour prompt caching cut this model's cost roughly in half, from about $115 of equivalent uncached traffic. All figures come from the providers' own reported usage, not estimates.
Limits
A benchmark's limits are part of its result, so here they are in full.
- Depth. About 135 graded runs per model gives a margin of roughly ±8 points; the pooled rate is ±3. No ranking between models is supported. A published per-model rate would want about 600 runs per model.
- One agent setup. Direct HTTP tool calls only, with no agent framework on top. Scaffolding is known to matter: an earlier internal measurement found a 4.2× cost difference and a large accuracy difference from the scaffold alone. This test does not vary it.
- Integration style is controlled, not eliminated. Our tool layer presents plain JSON Schema, and models tuned for strict structured outputs behave differently against it (defect five). The corrected numbers strip the resulting padding; they do not make the two styles equivalent.
- Two task types are not quotable on their own. Delayed consequence and cross-system composite sit near the floor for every model, and their answer keys are not yet validated.
- Refusing and doing nothing look the same. By state alone, a correct refusal and an agent that did nothing are identical. The null-agent floor, about 22%, is the control for this. Only the transcript shows the difference, and reading it is deliberately left out of grading.
- The 100% comparison is a check that trusts the claim. That is a real and common pattern, but it is not an AI judge reading full transcripts. That comparison is built but has not yet been run against these records.
- Copies, not production. The worlds reproduce each system's real routes, error messages and rules, and one is verified against the live API. They are not the live systems.
- Mixed sources across models, as described in the re-run section above.
Reproducing
Every record carries its code commit, seed hash, random seed and an artifact path. Task selection is deterministic, every run gets a fresh world from a content-hashed seed, and replay is byte-identical. Results measured on different code are never compared.
The transcript is what the agent believes. The system state is what happened. On these 824 runs they disagreed 303 times, and the agent was never the one that was right.