- Why it matters if true
- Most dependency-upgrade tooling checks whether a package still builds, not whether the new version silently changes behavior at the call sites your code actually exercises. If an agent can reason about usage-scoped impact instead of just diffing changelogs, upgrade review stops being a manual read-every-line task.
- Instrument
- TODO: fill in — name the harness/benchmark used here
- Kill threshold
- TODO: fill in — state the threshold as pre-registered, or say plainly none was set
- Results
- Loud bugs (crashes, obvious signature breaks) are caught reliably. Quiet bugs (behavior changes that don’t error) are caught at a rate ranging from 18% to 100% depending on how the input/test generation was set up — a wide enough spread that the number alone is not yet a claim worth publishing on its own.
- What I don’t trust yet
- The 18–100% range is doing the work of a red flag, not a result: it means the measurement is sensitive to input generation choices that haven’t been controlled for yet. TODO: fill in once that’s narrowed