The Commit Message That Matches the Diff, Benchmarked

A commit message is a claim about a diff. “add rate limiting to login” says the diff added rate limiting to login — and either it did, or it also touched three other files the message never mentions, or it “improves performance” nobody measured. You can’t tell which by reading the message. You have to read the diff, and almost nobody does. We run a directory that bench-tests Claude skills for a living, so we did the boring thing: we made Claude write commit messages for 15 real open-source diffs, twice each, and checked every message against the actual staged changes.

The result is commit-discipline, and it ships with the before/after numbers attached: diff coverage went from 83% to 97%, subjects fit inside 50 characters on 15 of 15 diffs (up from 6 of 15), and fabricated claims dropped from 1 to 0. It’s free and MIT-licensed: github.com/Skillproofdev/commit-discipline.

The gap: everyone checks the format, nobody checks the truth

Before writing a line we surveyed 80 commit-focused skills in our 16,682-skill index, plus the standalone commit skills published for Claude Code. The format layer is a crowded, solved space. Conventional Commits compliance is universal. Imperative mood, 50-character subjects, BREAKING CHANGE: footers — common, well-documented, table stakes. If all you want is type(scope): subject shaped correctly, dozens of skills already do that.

What none of them enforce is the layer underneath: whether the message is true against the diff. Does it account for every logical change that’s staged? Does it invent nothing — no guessed motive, no unmeasured “improves performance,” no change described that isn’t actually there? That’s the honesty layer, and it was uncontested in the field. Every competitor either runs git diff --cached as an unenforced suggestion or openly works from file lists — filenames, which tell you where something changed but never what or why. commit-discipline makes reading the full diff Rule 1 and bans the filename shortcut outright.

Eight rules, four of them nobody else codifies

The skill is a set of hard rules (full SKILL.md). The familiar parts are done strictly: a type earned from what the diff does to behavior, a scope that must be derivable from the paths (never invented), a ≤50-char imperative subject, breaking changes in a footer. The parts nobody else enforces:

1. Diff-coverage completeness. After drafting, walk the diff against the message: every logical change accounted for, nothing described that isn’t staged. Hallucination control for commit messages.
2. Read the full staged diff, never filenames. Rule 1, with the shortcut banned. The message describes the diff, so the diff — all of it — is the input.
3. A concrete banned-vagueness list. update, fix stuff, improve, misc, cleanup with no object, wip, address feedback — blocked as load-bearing words, each with a replacement pattern (update deps → bump axios 1.6→1.7).
4. An output contract. The deliverable is the message alone in one code block — no preamble, no diff play-by-play, no surprise Co-Authored-By footer.

Plus a body that explains why and impact instead of restating the diff (with a deletion test: if a line could be reconstructed by reading the diff, cut it), and a split recommendation for multi-concern diffs — concrete file boundaries and a draft subject per commit — instead of one umbrella message papering over two changes.

The benchmark: 15 real diffs, original messages stripped

We pulled 15 fixtures from curl, redis, express, fastapi, eslint, django, rust-analyzer and astro at pinned SHAs: features, bug fixes, refactors, two genuine breaking changes, two planted multi-concern diffs, docs and CI chores, and 400-plus-line many-file commits. Each diff went to two Claude Sonnet agents with identical prompts. The only difference: one read this SKILL.md first, one didn’t. We scored three ways — mechanical Conventional-Commits compliance by script, diff coverage against a pre-registered gold change list, and blind A/B preference judging.

Metric	Baseline	With skill
Spec compliance (7 mechanical checks, mean)	91.4%	98.1%
— subject ≤ 50 chars	40% (6/15)	100% (15/15)
Diff coverage (vs gold change lists, mean)	83.2%	96.7%
Fabricated claims (total across 15)	1	0
Breaking-change recall (2 fixtures)	2/2	2/2
Split detection (2 planted multi-concern diffs)	0/2	2/2
Blind A/B preference (skill vs baseline)	—	9 win / 4 loss / 2 tie

Two things stand out. First, mechanical compliance was already strong at baseline — correct types, imperative mood, breaking footers and non-vague verbs came out of the box. The entire format gap was subject length: the baseline overran 50 characters on 9 of 15 diffs; the skill never did. Second, the real separation is coverage. The baseline routinely dropped secondary changes — the added tests, the changelog entry, the second concern hiding in a “single” diff — while the skill accounted for them. That 83%→97% jump is where most of the preference wins came from, and it’s the thing no format-checker can give you.

Split detection is the cleanest illustration. On the two planted multi-concern diffs (t05, t12), the baseline wrote one umbrella message each; the skill proposed the split both times. On t07 — a curl commit removing TLS-SRP — the skill even caught an unrelated six-setopt hunk the baseline silently absorbed into its message, which was the baseline’s one fabrication of the run: an invented removal rationale (“virtually unused”) the diff didn’t support. The skill flagged that hunk as a question to the user instead of guessing.

Where the skill lost — published anyway

Our methodology requires the losses next to the wins, and this was not a clean sweep.

The baseline beat the skill on three fixtures. On t03 (redis), t13 (rust-analyzer), and t15 (fastapi) — all tightly-scoped diffs — the baseline caught a specific nugget the skill glossed over: a widened &mut dyn SourceDatabase trait bound (t13), a _build_dependant dedup extraction (t15), and one of two diff-counting algorithms (t03). The skill’s completeness pressure is real but not absolute; on small single-file diffs the baseline matches or beats it.

The skill’s split discipline over-fired once. On t14 it split a two-line CI-plus-dependency commit — add Node 26 to the matrix, bump mocha — into two, defensible by the letter of “one commit, one concern” but something most reviewers would keep as a single ci: commit. Three correct splits, one over-split. Counted as a preference loss and flagged.

And the caveat that matters most: n=15 is directional, not statistically significant, and the coverage and preference judging was done by a same-model-family judge — no cross-family GPT-class judge was configured on the test machine. Preference in particular carries self-preference risk: the writer and the judge share a model family. We shuffled the labels with a pre-registered seed and unblinded after scoring, but we’re not going to pretend a 9-4-2 preference tally from an in-family judge is a verdict. It’s a signal. The full protocol, per-fixture scores, and the judging-method disclosure are in bench/results/verdict.md.

Install

git clone https://github.com/Skillproofdev/commit-discipline ~/.claude/skills/commit-discipline

Restart Claude Code. It triggers on “commit this,” “write a commit message,” committing staged changes, and message-review or cleanup requests — and stays out of the way for git operations that aren’t about messages: branching, rebasing, conflict resolution. It joins our discipline series alongside token-discipline, which cuts what a session costs, and research-discipline, which cuts what an answer gets wrong. This one cuts what a commit message misses.

FAQ

Doesn’t Claude already write decent commit messages? For format, yes — that was the surprise in the benchmark. The baseline nailed type, mood, and breaking footers out of the box. Where it fell short was coverage: it dropped secondary changes on more than half the multi-concern and multi-file diffs, and it overran the 50-character subject on 9 of 15. The skill’s job is that gap, not the format everyone already gets right.

Is this just a Conventional Commits linter? No, and that’s the point. Conventional Commits compliance is a crowded, solved space — dozens of skills do it. commit-discipline enforces the layer underneath: that the message accounts for every logical change in the diff and fabricates nothing. Format is table stakes here; honesty against the diff is the product.

Will it commit or push for me? No. Writing the message is the job; running git commit is a separate instruction the skill won’t take on its own. It also won’t add Co-Authored-By or “Generated with” footers unless your project’s log or your own instructions show you want them.

Should I trust the 9-4-2 preference number? Treat it as directional, not a verdict. It was judged by a same-model-family agent with the labels blinded, because no cross-family judge was available on the test machine — so it carries self-preference risk, and n=15 isn’t statistically significant. The coverage numbers (83%→97%) rest on a pre-registered gold change list and are the more load-bearing result; the three fixtures the skill lost are named above.