Code Review That Proves Every Bug It Flags — Benchmarked

Ask an AI to review your PR and it hands you a tidy checklist result: injection checked, secrets checked, auth checked, looks good. It reads like diligence. Whether it actually found anything is a separate property — and the checklist is exactly what hides the miss. We run a directory that bench-tests Claude skills for a living, and when we pointed our own benchmark at a review skill we’d already rated pass, it walked past a hardcoded secret that the unguided baseline caught. The list had no line for that spot, so the reviewer never looked.

That miss is why review-discipline exists, and why it exists in its second version. It’s free and MIT-licensed: github.com/Skillproofdev/review-discipline. Every finding it reports has to name a concrete input that reaches the code and produces a wrong outcome — or it ships downgraded to [suspicion], never asserted as fact. On our seeded-bug bench it catches 18.5 of 21 planted bugs with zero false positives and a demonstrated failure path on ~100% of findings.

The gap: every review skill is a checklist, and checklists tunnel-vision

Before writing a line we surveyed 321 crawled review skills plus the standalone field — including Anthropic’s own code-review plugin. Three mechanics showed up in none of them as enforceable single-agent rules. All three trace back to the same failure we watched happen in our bench.

A checklist tells the reviewer what to look for. That’s also a list of what to look past. When the categories don’t include the bug, the bug survives — and worse, the review still comes back green, because every box that exists got ticked. The two ways a review fails are reporting undemonstrated guesses as fact (noise) and quietly skipping a real bug because it’s small or because a scarier one absorbed all the attention (missed coverage). A checklist is structurally good at producing both.

Anthropic’s plugin fights the noise half with parallel scorer agents voting a confidence number. That’s a real mechanism, but it needs multiple agents and it doesn’t address the coverage half at all. Nobody ships the thing that inverts the checklist’s shape: hunt everything before you reach for any category list, prove or label each finding second, and drop only what you can actually refute — in one agent, in one installable file.

Eight rules, three of them nobody else enforces

The skill is a set of enforceable rules (full SKILL.md). The familiar parts are there: P0–P3 severity with real definitions instead of vibes, diff-scope discipline (review the change, not the codebase), missing-test callouts tied to the changed behavior, and zero praise filler. The parts nobody else enforces:

1. No demonstrated failure path, no finding. Every finding must name a concrete input or state that reaches the flagged code and produces a wrong outcome. Can’t construct one? It ships as [suspicion], ranked below every proven finding — never as fact. An undemonstrated claim asserted as a bug is the skill’s one forbidden output.
2. Two passes, open hunt first. Read the change as an attacker with no category list in hand and record every anomaly — no severity floor, a thing that only smells slightly off still gets written down. Only then sweep the standard checklist (secrets, injection, boundaries, arithmetic, concurrency…) for anything the open hunt missed. Every competitor is the checklist; here it runs second, on purpose.
3. Kill your own finding before you report it. One honest refutation attempt per finding — upstream guards, intentional behavior, existing test coverage, reachability — and the report says what was checked. Refutation is the only gate allowed to drop a finding; “I couldn’t be bothered to prove it” becomes a [suspicion], not a silent discard. Findings that genuinely die, die silently.

The load-bearing rule is the order: breadth before depth. Collect every anomaly with no severity floor before you verify a single one, so the deep-dive on one bug can never truncate the hunt for the rest. That sounds obvious. It is also exactly the rule our first version didn’t have — and the reason it lost.

The honest benchmark (negative results included)

Seeded-bugs protocol: real code samples (~200–400 lines each, TypeScript / Python / JS) seeded with documented bugs of known severity — logic, security, edge-case, concurrency — with ground truth committed before any run. Real, intact code stays in each sample to measure false positives. Same prompts, same model; the only variable is whether the agent reads the SKILL.md first. Full methodology: skillproof.dev/methodology.

4 samples, 21 seeded bugs, 12 intentionally-healthy false-positive traps. The interesting column isn’t base-vs-skill — it’s v1 vs v2, because our own bench is what forced the rebuild.

Metric	Base (no skill)	Skill v1	Skill v2
Seeded bugs caught (of 21)	17.0 (81%)	16.5 (79%)	18.5 (88%)
P0 catch (exploitable / data-loss)	3/3	3/3	3/3
P1 catch (wrong behavior, realistic path)	7/7	7/7	7/7
P2 catch (edge / arithmetic paths)	6/7	4/7	7/7
Hardcoded-secret probe	caught	caught	caught
False-positive rate	5.6% (1 FP)	0%	0%
Findings with a demonstrated failure path	~63%	~100%	~100%
[suspicion] downgrades (honest hedges)	0	3	5
Praise / filler lines	present	none	none

Read the middle column and you can see the failure that named this skill. v1 scored 16.5 — below the unguided baseline’s 17. It had the failure-path discipline (it cut false positives to zero and demonstrated ~100% of its findings), but it was worse at finding bugs, because it locked onto the scary P0s in one sample and never swept the quiet money-rounding lines. It tunnel-visioned. Our public bench of a skill we’d rated pass is what caught it: v1 missed two real P2s — a prorate /30 logic error and an int(amount × 100) truncation — that the baseline picked up just by reading linearly.

The fix was the breadth-before-depth rule. v2 collects every anomaly with no severity floor before verifying any of them. Result: P2 went from 4/7 to a clean 7/7 (it even recovered an empty-CSV StopIteration that both base and v1 missed), total recall climbed to 18.5 — past base’s 17 — and the discipline held: still zero false positives, still ~100% demonstrated paths, and more honest [suspicion] hedges (5 vs 3) rather than fewer. Full 3-way adjudication is in bench/results/verdict.md.

One myth the bench also killed, plainly: the checklist-tunnel-vision-on-secrets hypothesis that motivated the whole project did not reproduce in the seeded run — all three arms caught the hardcoded-secret probe (S4-B1). The original miss was real and it’s what taught us the shape of the problem; the seeded bench just showed the secret itself isn’t where breadth pays off. The money-rounding edge cases are. We’re reporting the mechanism that actually moved the numbers, not the one that made the better origin story.

Where v2 lost — published anyway

Our methodology requires the losses next to the wins. v2 is the best arm on everything that blocks a merge, but it is not a strict superset of v1, and we won’t pretend otherwise.

Chasing exhaustive breadth, v2 stopped drilling into one function (_parse_tags) and walked past a subtle literal_eval deep-nesting suspicion (S2-B5) that v1’s depth-first pass had uniquely caught. So on the P3 / outside-checklist axis, v2 actually regressed — from v1’s 2.5/4 down to 1.5/4. Net it’s a good trade (+2 mediums for −1 subtle P3), but it’s a real regression on that one axis, not clean dominance. The ideal reviewer is v2’s breadth plus v1’s willingness to keep drilling one more level on quiet-looking code — and we’d rather tell you that than round it off.

There’s also a bug no arm caught: a coupon-expiry strict-< tie (S1-B6), missed by base, v1, and v2 alike. The skill cuts misses; it doesn’t make Claude infallible.

Install

git clone https://github.com/Skillproofdev/review-discipline ~/.claude/skills/review-discipline

Restart Claude Code. One command — the repo is the skill. It triggers on “review this PR/diff”, “check this before merge”, “find bugs in”, and pre-merge checks — and stays out of the way for feature writing, style-only linting, and prose review. It joins our discipline series: token-discipline cuts what an agent costs, research-discipline cuts what it gets wrong on facts, and this one cuts what a review misses.

FAQ

How is this different from Anthropic’s code-review plugin? The plugin filters false positives by having parallel scorer agents vote a confidence number — effective, but it needs multiple agents and it only addresses the noise problem. review-discipline filters with a demonstrated failure path in a single agent, in one installable file, and it also attacks the coverage problem the plugin doesn’t touch: the open-hunt-before-checklist rule that lifted our own P2 recall from 4/7 to 7/7.

Won’t “prove every finding” make it miss things it can’t demonstrate? No — that’s the [suspicion] mechanism. A real bug you can’t build a failure path for (needs runtime state you can’t see, or an external system) still gets reported, marked [suspicion] and ranked below the confirmed findings, with one line on what’s missing. Downgrading confidence is the tool; dropping is not. In the bench, v2 emitted 5 such honest hedges rather than swallow them.

Does it review the whole codebase or just my diff? Just the change. Findings must be caused or activated by this diff; pre-existing problems go in a short out-of-scope note, not the ranked list. The one exception is a pre-existing P0 — a live secret or active vulnerability — which is always flagged prominently. That carve-out exists precisely because of the origin-story miss.

Is 18.5/21 good enough to skip human review? No. It catches every P0 and P1 merge-blocker in our bench and beats an unguided baseline on total recall with zero false positives — which makes it a strong first reviewer that never rubber-stamps and always names what it attacked. But it missed one seeded bug entirely and traded away a subtle P3 for breadth, both documented above. Use it to make sure the obvious and the quiet-arithmetic bugs don’t reach a human; keep the human for the last level of depth.