The Skill Bench: systematic-debugging vs. baseline

Part 3 of the Skill Bench asks a narrower question than the first two: not “does a skill help,” but “does a skill built for one job beat an unaided model doing that exact job.” Debugging is the cleanest test we have for that, because we can plant the bugs ourselves, know precisely what’s wrong, and check the report against ground truth line by line.

So we wrote a canvas Snake game, broke it on purpose in three specific ways, and ran the same broken game and the same complaint through two identical Claude Sonnet sessions. One had the systematic-debugging skill installed. One didn’t. Same prompt, same model, same bug. Only the skill differs.

The result was not the clean win we expected going in, and that’s the part worth reading.

The trap we built

We started from a working Snake implementation, then planted three bugs deliberately:

1. Inverted direction vectors. ArrowUp and ArrowDown were wired to the vectors you’d use in normal Cartesian coordinates, not canvas coordinates, where y increases downward. Press up, the snake goes down.
2. Food spawning in pixel space instead of grid space. spawnFood used cv.width (400 pixels) as the bound for a coordinate that was supposed to be a grid index in a 20-cell grid (GRID). Multiply a grid step by a 400-wide range and you get food coordinates that land off the visible board roughly 95% of the time.
3. Score incrementing every tick. score++ sat in the main game loop instead of inside the “snake ate food” branch, so the score climbed on every frame regardless of what the snake actually did.

We did not tell either Claude session what was wrong. We gave both arms the same player-style bug report, phrased the way an actual annoyed tester would phrase it: “controls feel wrong, food never appears, score climbs by itself.” Then we asked each session to find and fix everything causing that.

Three bugs, one honest complaint, two Claudes. Here’s what came back.

What each arm reported

Baseline (no skill)

The baseline session worked through the code without any imposed method, reading the input handler, the food spawner, and the game loop in turn. It landed on all three planted bugs:

• Direction vectors were flipped for the y-axis relative to how canvas rendering treats “down.”
• spawnFood multiplied cv.width where it should have used the grid constant, producing pixel-scale coordinates in a cell-indexed game.
• The score increment sat outside the collision-with-food check, so it fired unconditionally every frame.

Then it kept reading, and found a fourth issue nobody had planted: the self-collision check compared the snake’s head against the tail cell before the tail had been removed for that frame. When the snake moves into the square its own tail is vacating, the check still sees that square as occupied and calls it a collision. That’s a real, well-known Snake edge case. It existed in our code because we wrote the collision logic before we planted anything, and the baseline just happened to read far enough to hit it.

Skill arm (systematic-debugging installed)

The skill arm read obra/superpowers’ systematic-debugging SKILL.md first (a skill we’ve separately scored 9.6/10 for hypothesis-driven root-causing), then applied its method: form a hypothesis for each symptom, test it against the code, confirm before touching anything.

It found the same three planted bugs, with tighter, more precise root-cause language:

• “Direction inversion traced to canvas y-down convention: ArrowUp maps to {x:0,y:-1} assuming Cartesian y-up, but the canvas renders y increasing downward, so up and down are swapped.”
• “Food spawn uses pixel-space bound (cv.width = 400) where the calculation should use grid-space bound (GRID = 20). Confirmed by unit mismatch: multiplying a random grid-scale fraction by a pixel-scale bound puts ~95% of spawns outside canvas.width.”
• “Score increments unconditionally in the tick loop instead of being gated on the food-eaten branch. Confirmed by tracing the loop body: score++ executes before the collision check runs, on every frame.”

It did not find the fourth bug. The self-collision issue never surfaced, because the reported symptoms (wrong controls, missing food, runaway score) didn’t point at collision logic, and the skill’s hypothesis-driven method stays anchored to the symptoms it was given. It closed out the report confident all issues were resolved, and by the letter of the complaint, they were.

The fourth-bug plot twist

This is the twist we verified by hand, re-reading both diffs against the actual code: the skill arm was more precise about the bugs it was asked to find, and the baseline arm found one more real bug that nobody asked about.

That’s not a knock on systematic-debugging as a skill. Hypothesis-driven debugging is built to do exactly what it did here: take a reported symptom, narrow to a testable cause, confirm before fixing, and stop once the symptom is explained. That discipline is precisely the point when you’re staring at a production incident with a stack trace and a clock running. It is also, by construction, symptom-scoped. It doesn’t wander into code that isn’t implicated by the complaint.

The baseline session had no such scope. It read more of the file than it strictly needed to, and reading more of the file is how you stumble into a bug nobody mentioned. Free-roaming exploration is inefficient by design, and here the inefficiency paid for itself.

The honest reading: neither arm is “better” in general. One arm is precise and cheap in attention, anchored to what it was told. The other is unfocused and, this one time, thorough enough to catch something the ticket didn’t mention. If your bug report is complete, the skill arm’s report is the one you want to read. If your bug report might be missing something, you want the second pair of eyes the baseline gave you for free.

The numbers

	Baseline (no skill)	Skill arm (systematic-debugging)
Bugs found (of 3 planted)	3 / 3	3 / 3
Unplanted real bug found	Yes (tail-vacate collision)	No
Root-cause precision	Correct, less formalized	Correct, named mechanism per bug
Report structure	Ad hoc	Hypothesis → test → confirm, per bug
Tokens used	50,338	58,357
Token delta	—	+16%

The skill cost 16% more tokens for a report that reads better and pins down mechanism more precisely, but it did not out-find a baseline that was simply allowed to keep reading. That’s the finding we didn’t expect walking in, and it’s the reason we’re publishing the raw comparison instead of a verdict that flatters the skill.

When you want systematic-debugging

The Snake bench understates the skill’s real value, because a planted-bug game with a full source file in view is close to the best case for an unaided model: everything relevant fits in one read. Production debugging rarely looks like that.

Where hypothesis-driven root-causing earns its keep is the bug that comes back. In our own catalog notes on systematic-debugging, one test case was a race condition that Claude had already “fixed” three separate times in the same codebase, each time patching a plausible-looking guess instead of the actual cause. The skill’s discipline (state the hypothesis, design a test that would falsify it, don’t touch the code until the test confirms it) was what finally pinned the real interaction instead of producing a fourth guess. That’s the shape of bug systematic-debugging is for: recurring, evasive, guessed-at three times already, in a codebase too large to read end to end on a whim.

It’s also the right tool the moment you’re triaging a production incident. You have a symptom, a clock, and no appetite for a model wandering off to read unrelated modules. Scope to the report, confirm the cause, ship the fix. That’s a feature, not a limitation, in exactly that setting.

When a free sweep wins

Our Snake result argues for the opposite case too: when you suspect the bug report is incomplete, or when you’re doing a pre-release pass rather than chasing one reported symptom, an unconstrained read of the code is doing something a hypothesis-driven method structurally won’t. It’s looking at code nobody complained about yet.

That’s the same shape as a manual code review versus a targeted incident response. Both have a place. The mistake is assuming a debugging skill replaces a review pass rather than sitting alongside one. If you want the general review-pass angle on this, see our take on why skills don’t always trigger the way you expect and how we structure these tests.

Reproduce it yourself

The bench is small enough to rebuild in twenty minutes if you want to check our numbers rather than take them on faith. Write a canvas Snake game (grid movement, a food spawner, a score counter, a game loop) and plant these three defects:

1. Wire ArrowUp/ArrowDown to Cartesian-style vectors (y: 1 for up, y: -1 for down) instead of canvas-style vectors, where increasing y moves down the screen.
2. In your food-spawn function, multiply your random fraction by the canvas’s pixel width instead of your grid cell count, so the resulting coordinate is a pixel value being treated as a grid index.
3. Move the score increment out of the “did the head just land on food” branch and into the unconditional body of the game loop.

Give a Claude session only the player complaint (“controls feel wrong, food never appears, score climbs by itself”), never the bug list, and see what comes back. Then read your own collision logic for the tail-vacate case: check the head’s next position against the tail’s current cell before the tail moves, and see if it’s there too. Ours was, without us planting it.

FAQ

Does the systematic-debugging skill make Claude worse at finding bugs?

No. It found every planted bug with more precise root causes than the baseline. What it didn’t do is exceed the scope of the reported symptoms, which is what let the baseline stumble onto a fourth, unplanted bug. Scope discipline is the skill’s design intent, not a defect.

Why did the baseline find a bug the skill missed?

The baseline had no hypothesis to stay anchored to, so it read more of the codebase than the reported symptoms strictly required. Reading more code is how you find things nobody asked about. It’s an inefficient strategy that happened to pay off once, not a general advantage.

Is a 16% token increase worth it for a debugging skill?

Depends on the bug. For a recurring, hard-to-pin issue, the discipline is worth far more than 16% in tokens, because the alternative is Claude guessing at the same cause repeatedly, as it did in our own catalog test before we applied this skill. For a bug that’s already well-scoped and shallow, the overhead buys you a cleaner report and not much else.

Should I use systematic-debugging for every bug fix?

Not every one. Reach for it when a bug is recurring, when a previous fix attempt didn’t stick, or when you’re triaging a live incident and need to stay scoped to the reported symptom. For a first pass on a fresh bug report, or when you want a broader sweep that might catch adjacent issues, a plain debugging session, or a dedicated review pass, can cover ground the skill won’t. See our best coding skills for how we rank debugging and review skills against each other.

---

The Skill Bench, part 3 of 4. Read the other controlled comparisons: part 1, the landing page build, part 2, the Telegram bot, and part 4, auditing Google zx.