QA-Bench v0: measuring how AI models handle code verification
Given a real pull request on a production codebase, can a model find every affected user flow and catch what breaks? We put a purpose-built agent against the frontier models across 35 PRs on four production-scale repos.
Background
Code agents ship fast. In 2025, the median time from git diff to merged PR shrank from days to minutes. The quality gap followed. Coding agents have no instinct for user-visible breakage — they optimize for tests that pass and reviewers that merge, not for the UX flows that quietly shatter downstream.
We wanted a number. Not a vibe. A reproducible score that answers: for a given PR on a real codebase, does this model find the breaks before users do?
Methodology
We selected 35 merged PRs across four production-scale repos spanning e-commerce, SaaS dashboards, developer tooling, and a payment API. Each PR was scored by three senior engineers who documented every user-visible regression introduced — including ones that made it past CI.
Each model was given: the diff, the repo context window, and a task: enumerate every user flow this PR could break, then verify which ones actually break. No scaffolding, no test harness, no orchestration layer — just the model and the codebase.
RICERY ran its standard adversarial loop: recon → attack → break → verify, with full access to the staging environment. The score reflects the percentage of engineer-documented breaks that each system surfaced.
Score breakdown
Accuracy decomposed by task type across the 35 PRs:
| Category | RICERY | GPT-5.4 | Claude Code | Sonnet 4.6 |
|---|---|---|---|---|
| Flow discovery | 91.2% | 85.4% | 82.1% | 76.8% |
| Break detection | 87.4% | 81% | 79.3% | 74.5% |
| Root cause accuracy | 79.6% | 76.2% | 73.1% | 68.4% |
| Regression coverage | 74.2% | 78.3% | 77.6% | 73.1% |
What the gap looks like
The largest delta between RICERY and frontier models is in flow discovery: finding the full set of user-visible paths affected by a change. Frontier models consistently found the obvious flows — the ones a careful reviewer would also spot in the diff. They missed the transitive ones: a billing calculation refactored for correctness that broke a quarterly-report export three layers up the call stack.
Root cause accuracy tells a similar story. Models that found a break frequently misattributed it — pointing at the symptom rather than the upstream invariant that changed. That matters for regression arming: a test written against the wrong cause doesn't rerun when the real cause is touched again.
The one category where RICERY trails is regression coverage: GPT-5.4 and Claude Code both generate broader test scaffolding. We're measuring whether RICERY's targeted regression harness (one precise rerun per break) outperforms broad coverage over time — v1 of this bench will track that.
Limitations and next steps
35 PRs across 4 repos is a starting point, not a verdict. The repos were selected to cover surface diversity (UI, API, MCP, background jobs) rather than for size or team culture. We're expanding to 200 PRs across 12 repos for v1, with blind scoring to reduce evaluator bias.
The benchmark is open. If you'd like your repo included in v1 — or if you want to run it against your own model — reach out.