The Skill Bench, part 2: Claude TDD skill vs no skill

Part 1 of this series pitted a skill against a blank prompt on a scaffolding task and got a lopsided win. Part 2 does not. We built the same Telegram bot twice with Claude Sonnet, gave one run the highest-scored test-driven-development skill in our catalog, and the skill added tokens without adding a test that mattered. We’re printing that result as-is, because a benchmark you only publish when the skill wins isn’t a benchmark.

Setup and methodology

This is N=1. One task, one model, one skill, run once per arm. That’s not enough data to claim a percentage difference in quality, and we’re not going to pretend otherwise. What it’s enough for is showing what actually happens in a single session, line by line, which is a thing most skill marketing never shows you at all.

The task: build a Telegram todo bot on aiogram v3 with four commands, /add, /list, /done, /delete, backed by in-memory storage scoped per user. The brief asked for a clean split: storage.py holding pure logic with no Telegram imports, bot.py wiring that logic to aiogram handlers, and test_storage.py covering the storage layer. Run pytest until it’s green, then stop.

Both arms got the identical prompt, the identical model (Claude Sonnet), and the identical repo scaffold to start from. The only variable: the skill arm had obra/superpowers’ test-driven-development SKILL.md installed, the same skill that scored 9.6 out of 10 in our catalog for enforcing a strict red-green-refactor loop and refusing to let a session mark work done while a test is failing. The baseline arm had nothing installed beyond default Claude Code behavior. Full method for how we script and log these runs is in how we test Claude skills, and the broader scoring rubric lives at our methodology page.

What both arms shipped

Both arms finished. Both hit a fully green pytest run. Both produced the three-file split the brief asked for, with storage logic isolated from the aiogram handler code. On the surface, this looks like a tie, and if you stopped reading at “both went green,” you’d conclude the skill made no difference and move on.

Neither bot.py file did anything surprising. Each wired the four commands to aiogram’s Router and Message handlers, parsed the todo text or index out of the command arguments, and called straight into the storage layer for the actual work. That’s exactly what the brief asked for: keep the bot file thin, keep the logic testable. On the aiogram wiring, the two sessions converged almost completely, which is itself a useful data point. Where they diverged was entirely on the storage side, in what each considered worth a test.

The difference shows up in what each arm decided was worth testing, and in what it cost to get there.

The numbers

	Baseline (no skill)	Skill (test-driven-development)	Delta
Total tokens	48,536	52,480	+8%
Tests written	23	13	-10
Exception-path tests	9	5	-4
Final pytest status	Green	Green	Tie

The skill arm used more tokens to produce a smaller test suite with less exception coverage. That’s the whole result. No hidden asterisk, no “but if you look at code quality instead of test count.” The baseline session, running with no process scaffolding at all, wrote almost twice as many exception-path cases.

Reading the test suites: 23 vs 13

Test count alone is a weak signal, so we read both suites line by line instead of trusting the header number.

The baseline’s 23 tests covered the four commands at the happy-path level, then kept going into edge cases nobody asked for by name: what happens when you mark an out-of-range index done, what a zero or negative index does, whether one user’s todo list leaks into another user’s, and whether deleting item 2 correctly shifts the indices of items 3 and 4 so /done 3 still points at the right task afterward. That last one is the kind of bug that survives a demo and then breaks in front of a real user the first time they delete something from the middle of a list. Nine of the 23 tests existed purely to poke at these exception paths.

The skill arm’s 13 tests covered the same four commands at the happy-path level, plus five exception-path cases: mostly out-of-range indices and one duplicate-add scenario. What’s missing relative to the baseline: no explicit per-user isolation test, and no test confirming index behavior after a delete shifts the list. The storage logic in the skill arm’s storage.py may well handle these cases correctly. Untested correct behavior and tested correct behavior aren’t the same claim, though, and the whole point of a test suite is closing that gap.

Neither suite is bad. Thirteen tests with five exception cases on a four-command bot is a defensible starting point by any normal engineering standard. The comparison only looks damning next to a baseline that, working from an identical prompt with no red-green scaffolding at all, wrote more tests, not fewer.

Why this doesn’t refute TDD

Here’s the thing a one-shot benchmark structurally cannot see: TDD’s argument was never “you’ll write more tests on your first try.” It’s about what happens over weeks of iteration, across the tenth feature added to a codebase the model didn’t write from scratch, at the point where a tired engineer (or an agent under time pressure) is tempted to ship with a red test and fix it “later.”

That’s a discipline claim, not a one-shot output claim, and this benchmark ran exactly one shot. We can’t measure discipline in a single session because discipline is what stops you from cutting a corner on session six, and there is no session six here.

We do have a trace of what that discipline looks like in practice, from our own test notes on the test-driven-development skill page: across a three-feature session, the skill refused to skip the red-green cycle even when the fix looked obvious and the temptation to jump straight to green was right there. It wrote the failing test first, watched it fail for the right reason, then wrote the minimum code to pass it, every time, across all three features. Nobody had to intervene and say “wait, write the test first.” That’s the behavior a discipline skill is supposed to buy, and it isn’t the same behavior as “writes more tests in one attempt on a small, well-scoped module.”

On a task this size and this well-specified, Claude Sonnet’s baseline judgment about what to test was already solid. The skill added an explicit process on top of judgment that didn’t need much correcting yet, and that process cost 8% more tokens without a matching quality win on this particular attempt. Both things can be true: TDD is worth having installed, and it didn’t help here.

There’s also a plainer explanation worth naming: writing a failing test, watching it fail, then writing the minimum code to pass it takes more back-and-forth than writing the implementation and a test for it in one pass. That overhead is the point of the discipline when the implementation is nontrivial or the model is prone to skipping ahead. On a four-command todo bot, the implementation was never in doubt, so the overhead bought process without buying a check on anything that was actually at risk of going wrong.

What this means if you’re buying skills

The uncomfortable part for us, specifically, is that we sell tested skills, and our own bench just showed a top-scored skill not winning a one-shot comparison against no skill at all. We’d rather you see that than a highlight reel.

The practical takeaway is to match the skill to the job, not to the score. A 9.6/10 catalog score means the skill does what it says reliably and doesn’t break your setup, not that it wins every benchmark on every task size. If your job is a small, well-specified module you’re building fresh, in one sitting, a strong model’s default judgment may already cover the exception paths you care about, and a process skill is overhead you’re paying for without a matching upside that session. If your job is a codebase you’ll touch for months, with multiple contributors and long stretches between sessions where corner-cutting compounds silently, that’s what a discipline skill like TDD is built to prevent, and a single-session benchmark was never going to capture that value in the first place.

Speed skills and discipline skills answer different questions. Read a skill’s description for which question it answers before you install it for the wrong job. We cover how to read that signal in our roundup of the best coding skills.

Reproduce it yourself

The task is small enough to rerun in an afternoon. Clone a bare aiogram v3 project, then run the identical prompt twice: once in a clean Claude Code session, once with obra/superpowers’ test-driven-development skill installed. Ask for /add /list /done /delete with per-user in-memory storage, a storage.py/bot.py split, and a pytest suite that runs green before you call it done. Log total tokens from each session’s usage summary, then diff the two test_storage.py files by hand: count assertions, and specifically flag anything touching negative indices, missing keys, per-user isolation, and index shifts after a delete. Those four categories are where we saw the gap, and they’re the ones worth checking on any todo-style CRUD app regardless of which skill you’re testing.

FAQ

Does this mean the TDD skill is bad?

No. It means a one-shot benchmark on a small, well-specified module isn’t the test that shows what TDD is for. The skill’s value is in preventing corner-cutting across a long session or a long project, which this benchmark, by design, didn’t run long enough to measure.

Why did the skill arm write fewer tests if it enforces a stricter process?

The red-green-refactor loop drives you to write a test for the behavior you’re about to implement, then implement it, then move to the next behavior. It doesn’t automatically prompt you to go back and add tests for edge cases nobody explicitly asked for, unless the session takes time to brainstorm them separately. The baseline arm, unconstrained by a fixed cycle, apparently spent more of its output on exactly that brainstorming.

Should I install a TDD skill for Claude Code?

If you’re working in a codebase you’ll return to repeatedly, especially with other contributors or long gaps between sessions, yes. It’s insurance against a specific failure mode: quietly shipping with a red test because the fix looked obvious. That failure mode doesn’t show up in a single afternoon’s benchmark, but it shows up in real projects.

What’s next in this series?

Part 3 of “The Skill Bench” looks at a debugging skill against an unmodified snake game bug hunt. Part 4 audits a Google zx automation script. Both follow the same rule as this one: same task, same model, one variable, numbers printed either way.

The Skill Bench series

Part 2 of 4. Read part 1: landing page build, part 3: debugging snake, and part 4: auditing a Google zx script.