An agent can attribute a regression to its cause from telemetry alone.
Partial- Why it matters if true
- The mid-market remediation thesis rests on “draft the change, human approves, we execute.” That loop depends on attribution. If an agent can’t reliably say what broke and why, a human re-enters at the most expensive point and the economics collapse into ordinary consulting.
- Instrument
causebench— [N] regressions injected into [corpus], telemetry graded at three tiers (rich / realistic / lower). The investigator sees only telemetry and must name the cause. Control runs contain no injected regression, to catch invented culprits.- Kill threshold
- TODO: fill in — or state plainly that none was pre-registered
- Results
Tier Correct attribution Rich 100% Realistic 93% Lower 30% false attribution on controls The controls are the problem. The failure wasn’t random: when logs were truncated, the agent treated the truncation itself as evidence — a cut-off log looks like something was hidden — and reasoned backward from “suspicious” to a specific cause, reporting it with the confidence of a real finding. That’s worse than a miss. A miss is visible. A confident wrong answer routes a human toward fixing a bug that doesn’t exist, which is exactly the labor cost the thesis was meant to remove.
- What I don’t trust yet
- The control arm had two distinct cases. A 30% false-attribution rate on that base is a smell, not a measurement. Phase 2 is a real control arm before anything else.