Changelogs You Can Trace to Real Commits — Benchmarked

A changelog is a factual claim about what a release does to its users. Read a good one and you can’t tell whether it’s true — every generator makes the entries pretty, and almost none makes them verifiable. LLM-written changelogs have documented failure modes: invented entries, hallucinated version numbers and dates, breaking changes buried or dropped, friendly rewrites that drift from what the code actually did. We run a directory that bench-tests Claude skills for a living, so we built the discipline layer that blocks each one — and then measured it against four real open-source releases.

The result is changelog-discipline, and this post publishes its numbers in full, including the ones where it lost. It’s free and MIT-licensed: github.com/Skillproofdev/changelog-discipline.

The gap: honesty and readability ship in different products

Before writing a line we surveyed 101 changelog and release-notes skills in our 16k-skill dataset, plus the standalone tooling — git-cliff, release-please, conventional-changelog. The two halves of a good changelog live in different products and never overlap.

Mechanical generators (git-cliff and friends) are traceable by construction: every line comes from a commit. But they only see conventional commits, so anything that doesn’t match feat:/fix: gets silently dropped, and they read like a parsed git log because that’s what they are. LLM generators write beautifully — user-impact language, clean grouping — but they verify nothing, so they invent entries, mint version numbers, and bury breaking changes when the release looks cleaner without them. Nobody enforces both honesty and readability, and nobody enforces breaking-change recall on top. That third property is the one that actually hurts when it’s missing: a dropped breaking change is the one unrecoverable changelog failure.

Eight rules, three that nobody else enforces

The skill is a set of hard rules (full SKILL.md). The familiar parts: Keep-a-Changelog grouping, user-impact phrasing that never widens a claim beyond the diff, versions and dates read from real tags rather than written from memory, and a mandatory self-audit pass before delivery. The parts nobody else enforces:

1. Derived from git, never from memory. The range is resolved and read — git log, plus diffs where subjects are vague — before a single entry is written. No repo access means no changelog, not a guess from “what we worked on.”
2. Every line traces to a real commit or PR. A trace map is built first; an entry that can’t point at a commit doesn’t ship, and every cited (#123) or (abc1234) must exist in the actual history.
3. Breaking changes are hunted, not awaited. Not just BREAKING CHANGE: footers — removed APIs, renamed flags, changed defaults found by reading the diff. They go first, marked BREAKING, with a one-line migration note.

The benchmark: four real releases, scored against human changelogs

We took four open-source repos with hand-curated changelogs as ground truth and picked one released tag range each, fetched at the pinned tags: Django 5.2→6.0 (the largest, 404 commits in the scored set), Tailwind CSS v4.0.0→v4.1.0, FastAPI 0.116.2→0.117.0 (a zero-breaking trap — its curated notes have no breaking section, so any entry presented as breaking is a fabrication), and curl 8.14.1→8.15.0. Two agents got the identical prompt and the same repo; the only difference was whether the agent read this SKILL.md first. We scored commit-coverage, fabricated entries (mechanically verified against real hashes and dates), breaking-change recall, format compliance, and blind readability.

Range	Arm	Traceable commit cov.	Fabricated	Breaking first?	Format (0–6)
Tailwind	base	0/164 (0%)	0	no	3
	skill	143/164 (87%)	0	yes	5
Django	base	23/404 (6%)	0	no	3
	skill	234/404 (58%)	0	yes	6
curl	base	22/278 (8%)	0	yes	3
	skill	59/278 (21%)	0	yes	6
FastAPI	base	12/18 (67%)	0	n/a	4
	skill	9/18 (50%)	0	n/a	6

Where discipline shows, the skill wins clearly. Traceability is its core thesis and it dominates: 87% vs 0% on Tailwind, 58% vs 6% on Django. The base agent writes fluent prose describing plenty of real changes — it simply cannot trace them back to commits, which is exactly the gap the skill exists to close. Format compliance was 30/36 across the skill’s four runs versus 13/36 for base; every base output used non-Keep-a-Changelog headings (“Features”, “Notable bug fixes”), dropped the ISO date, and tacked on a notes epilogue. And on breaking-change placement, across the three ranges that actually have breaking changes, the skill put them first and marked them BREAKING in all three; the base did so in one (curl).

Be honest: the headline metric was a tie

The one number we most wanted to move — fabricated entries — did not move. It was 0 for all eight outputs. A tie. Every #-citation resolved to a real reference (up to 195 of them in a single skill run), no invented versions or dates, and every spot-checked prose claim was commit-backed, including Django’s 11 CVEs. The reason is simple and we won’t dress it up: on this corpus the base agent was already disciplined enough not to invent entries, so the skill’s fabrication guarantee held but was never stress-tested. We report the tie rather than bury it.

Where the skill lost — published anyway

Our methodology requires the losses next to the wins, and there were real ones.

Base won pure readability on the two large repos. On curl and Django, the single blind rater preferred the base output — its curated-narrative style, with a dedicated CVE section on Django, reads better than the skill’s exhaustive 230-line Keep-a-Changelog block. The rater couldn’t see that the base version was untraceable and scattered its breaking changes; on readability alone, base’s prose won. The skill trades some readability on huge releases for structure and honesty, and that trade is visible.

Base even beat the skill on FastAPI traceable coverage — 67% vs 50%. This one is our favorite result, because the skill was right to lose it: base cited three extra internal commits (a mypy bump, a dependency-cache change, a pydantic.mypy tweak) that the skill correctly dropped as non-user-facing. The metric rewarded base for listing churn a user shouldn’t see. And on Tailwind breaking recall, base edged the skill 4/7 to 3/7 by phrasing a deprecation in prose that the skill filed under Added — phrasing luck on a corpus where no commit subject says “deprecate.”

Two honest caveats on the benchmark itself: the blind-preference score used 1 rater, not the 3 we specify, so it’s under-powered. And per-run token cost was not captured this run — the skill arm additionally pays to read SKILL.md, and we can’t yet tell you how much.

Install

git clone https://github.com/Skillproofdev/changelog-discipline ~/.claude/skills/changelog-discipline

Restart Claude Code. It triggers on “write a changelog”, “release notes for v2.3”, “update CHANGELOG.md”, and “what changed between 1.4 and 2.0” — and stays out of the way for blog posts, marketing copy, and commit-message authoring. It joins research-discipline, which cuts what a researched answer gets wrong, and token-discipline, which cuts what your context costs, in our benchmarked discipline series.

FAQ

How is this different from git-cliff or conventional-changelog? Those are traceable by construction but only see conventional commits, so non-conforming work is silently dropped, and they read like a parsed log. This skill reads the full range — diffs included, not just commit subjects — and writes user-impact language while still requiring every line to trace to a real commit. Traceability and readability, which no single tool in our survey enforced together.

Does it just dump the git log with prettier wording? No — the opposite. It maps every commit to either an entry or a conscious exclusion, hunts breaking changes in the diff, groups by Keep-a-Changelog headings, and puts breaking items first with a migration note. On FastAPI it correctly dropped three internal commits the base agent listed, which cost it a coverage point and was the right call.

Does it invent version numbers or dates? It’s built not to: the version heading is the real tag name and the date is the tag’s actual date, read via git in ISO-8601. Unreleased ranges go under ## [Unreleased] rather than getting a minted number. Across eight benchmark outputs, zero invented versions, dates, or PR numbers.

Should I trust the benchmark? Trust it as far as it goes, which we state plainly: the fabrication metric was a 0–0 tie because the base agent was already honest on this corpus, the readability score used one rater instead of three, and token cost wasn’t captured. The wins that are solid — traceability, format, breaking-change placement — are mechanically scored against the human changelogs and reproducible. The full verdict publishes every cell, including the losses.