The workday, graded — 30 tickets, claude-sonnet-5 (cold-vercel)
30-ticket shift · 5 runs per act · 1 row · graded by the world diff, never the transcript · 2026-09-05 11:58 UTC
| model | Act I — clear skies no storm | Act II — Stripe stormstripe-storm-shift | Act III — tickets stormtickets-storm |
|---|---|---|---|
cold-vercel · claude-sonnet-5 |
per-ticket p̄ 72% 64.3%–78.6%
flawless shifts 0/5 observed 0%
predicted p̄^30 0% (0%–0.1%)
dollars wrong $3,715.00 9 wrong · 5 missed · 24 overreach
reply ≠ world 4 4 claimed-not-done · 0 did-not-claimed · 0 amount
30 days later −$178.20 0/5 match the reference · 2 customers differ
measured 2.65M in · 140k out · $1.34/shift · 7 min/shift 678 turns · 675 tool calls
|
per-ticket p̄ 72.7% 65%–79.2%
flawless shifts 0/5 observed 0%
predicted p̄^30 0% (0%–0.1%)
dollars wrong $2,986.00 15 wrong · 5 missed · 19 overreach
reply ≠ world 2 2 claimed-not-done · 0 did-not-claimed · 0 amount
30 days later −$138.60 0/5 match the reference · 2 customers differ
measured 2.73M in · 142k out · $1.37/shift · 7 min/shift 709 turns · 708 tool calls · 35 errors
|
per-ticket p̄ 74% 66.4%–80.4%
flawless shifts 0/5 observed 0%
predicted p̄^30 0% (0%–0.1%)
dollars wrong $3,229.00 11 wrong · 5 missed · 22 overreach
reply ≠ world 3 3 claimed-not-done · 0 did-not-claimed · 0 amount
30 days later −$178.20 0/5 match the reference · 2 customers differ
measured 2.88M in · 147k out · $1.45/shift · 7 min/shift 731 turns · 721 tool calls · 45 errors
|
p̄ is ticket passes over tickets × runs with a 95% Wilson interval; predicted p̄^N raises the point estimate and both bounds to the shift length (independence assumed). "Flawless" means every ticket right and nothing else touched. "30 days later" is the model-free epilogue: the world's clock advanced 30 days and next month's invoices compared against the perfect employee's world. Cost is estimated from measured tokens at list prices.
Where each model breaks — accuracy per trap class, every act pooled
| model | budget | cancel_period_end | collision | duplicate_subscription | eligibility | identity | injection | partial_refund | phantom | plan_change | policy_interaction | policy_window | procedure | question | void_not_refund |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
cold-vercel · claude-sonnet-5 | 100% | 100% | 66.7% | 90% | 0% | 0% | 100% | 0% | 33.3% | 100% | 73.3% | 0% | 100% | 100% | 100% |
The cliff
At 72% per ticket (64.3%–78.6%), a 30-ticket shift is flawless 0% of the time; the interval allows 0%–0.1%. Observed 0/5.