The workday, graded — 30 tickets, claude-sonnet-5 (cold-langchain)
30-ticket shift · 5 runs per act · 1 row · graded by the world diff, never the transcript · 2026-09-05 10:18 UTC
| model | Act I — clear skies no storm | Act II — Stripe stormstripe-storm-shift | Act III — tickets stormtickets-storm |
|---|---|---|---|
cold-langchain · claude-sonnet-5 |
per-ticket p̄ 70.7% 62.9%–77.4%
flawless shifts 0/5 observed 0%
predicted p̄^30 0% (0%–0%)
dollars wrong $3,472.00 14 wrong · 6 missed · 21 overreach
reply ≠ world 4 3 claimed-not-done · 1 did-not-claimed · 0 amount
30 days later −$138.60 0/5 match the reference · 2 customers differ
measured 2.61M in · 141k out · $1.32/shift · 6 min/shift 673 turns · 677 tool calls · 1 errors
|
per-ticket p̄ 74% 66.4%–80.4%
flawless shifts 0/5 observed 0%
predicted p̄^30 0% (0%–0.1%)
dollars wrong $3,771.00 11 wrong · 5 missed · 21 overreach
reply ≠ world 3 3 claimed-not-done · 0 did-not-claimed · 0 amount
30 days later −$118.80 0/5 match the reference · 2 customers differ
measured 2.71M in · 138k out · $1.36/shift · 7 min/shift 710 turns · 701 tool calls · 36 errors
|
per-ticket p̄ 73.3% 65.7%–79.8%
flawless shifts 0/5 observed 0%
predicted p̄^30 0% (0%–0.1%)
dollars wrong $3,285.00 12 wrong · 5 missed · 22 overreach
reply ≠ world 2 2 claimed-not-done · 0 did-not-claimed · 0 amount
30 days later −$178.20 0/5 match the reference · 2 customers differ
measured 2.86M in · 146k out · $1.44/shift · 7 min/shift 719 turns · 712 tool calls · 45 errors
|
p̄ is ticket passes over tickets × runs with a 95% Wilson interval; predicted p̄^N raises the point estimate and both bounds to the shift length (independence assumed). "Flawless" means every ticket right and nothing else touched. "30 days later" is the model-free epilogue: the world's clock advanced 30 days and next month's invoices compared against the perfect employee's world. Cost is estimated from measured tokens at list prices.
Where each model breaks — accuracy per trap class, every act pooled
| model | budget | cancel_period_end | collision | duplicate_subscription | eligibility | identity | injection | partial_refund | phantom | plan_change | policy_interaction | policy_window | procedure | question | void_not_refund |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
cold-langchain · claude-sonnet-5 | 100% | 100% | 60% | 93.3% | 0% | 0% | 100% | 0% | 33.3% | 100% | 80% | 0% | 100% | 97.8% | 100% |
The cliff
At 70.7% per ticket (62.9%–77.4%), a 30-ticket shift is flawless 0% of the time; the interval allows 0%–0%. Observed 0/5.