A README Where Every Claim Traces to the Code

A generated README is a confidence trick. It reads clean, lists the right-sounding features, and hands you install commands that look correct — and you cannot tell by reading it which sentences are true. The failure mode is always the same: a feature the code doesn’t have, a command pattern-completed from training data instead of copied from the repo, and a suspicious absence of any limitations. We run a directory that bench-tests Claude skills for a living, so we built the skill we wanted and then measured whether it actually holds.

The result is readme-discipline: nine enforceable rules whose contract is that a claimed feature must be found in source before it may be claimed, every command is copied from the project’s real manifests and executed, and an honest limitations section is mandatory. It’s free and MIT-licensed: github.com/Skillproofdev/readme-discipline.

The gap: 101 README skills, none enforce truth

Before writing a rule we surveyed the landscape — 101 README-adjacent skills inside a 16,682-skill catalog, plus readme-ai and the Standard Readme spec. Every one of them optimizes something other than accuracy.

The biggest dedicated README skill (2.2k★) is a set of audience templates. The Standard Readme spec mandates section order but says nothing about whether the code blocks run. The dominant CLI generator produces polished output and then tells you, in its own docs, to review it for accuracy yourself — accuracy is offloaded to the human. The rest are badge maximizers and emoji-header decorators. Structure and looks are solved five times over. “Every claim traces to code” and “every example is verified runnable” showed up as enforceable rules in none of them.

That’s the whole niche this skill occupies: not making READMEs prettier, making them true.

Nine rules, three of them novel

The familiar parts are here — a fixed section order (what+why → install → quickstart → usage → config → limitations), single-audience calibration, no badge or emoji inflation. The three that nobody else enforces:

1. Fabrication ban with receipts. Every falsifiable claim — a feature, a flag, a supported platform — is grepped in source first. Found → you may claim it, in the code’s own vocabulary. Not found → it does not go in the README, not hedged, not “typically.” A verification log mapping claim → file:line ships with every README.
2. Commands are copied, never composed. Every fenced command comes from a real place: package.json scripts, a Makefile target, a CI step, the CLI’s own --help. Never npm run build because Node projects usually have one — check scripts first, then run it in a clean checkout.
3. Quickstart is a contract. From clone to one observable success in about 60 seconds, every step executable exactly as written, ending in a result the user can check — a URL that responds, a file that appears, output that matches a shown snippet.

Plus a mandatory, sourced limitations section (pulled from TODO/FIXME comments, error branches, empty stubs) and an audit-first mode that strips stale claims out of an existing README before touching its style.

The benchmark: 3 real repos, every command actually run

We picked three small OSS projects with thin READMEs and froze each at a commit SHA: a Node CLI (crossplatform-killport), a Python CLI/library (python-shaarli-client), and a Flask web service (csrgenerator.com). Per repo, two agents — one baseline, one reading the skill first — same model, same prompt, the only difference being whether the SKILL.md was in context. Every command in the table below was executed on a real machine (macOS 14, node v24.7.0, python3 3.13.1); commands whose runtime was absent (Docker) or that needed a live external service were excluded from the executability denominator and verified statically instead.

Metric (aggregate over 3 repos)	Baseline	Skill
Fabricated claims (lower better)	2	1
Command-executability	20/24 (83%)	15/17 (88%)
Section completeness	14/18	18/18
Blind preference	0/3	3/3
Mean trust (1–5)	3.67	4.67

The direction is consistent across all three repos on completeness and preference. The skill hit 6/6 sections every time; the baseline shipped no limitations section on any of the three repos — the single biggest completeness gap. And it was preferred on all three with a full point more trust.

Where the skill actually earned it — and where it slipped

The fabrication number deserves honesty, because it’s the headline you’d expect to be a landslide and it isn’t. 2 versus 1 is a narrow gap, and here’s why: both baseline agents leaned heavily on the repos’ existing accurate USAGE.md/docs/ prose, so they inherited correct content for free. The baseline’s two misses were exactly the failure this skill targets — a stale Python 3.4+ requirement contradicted by the project’s own test matrix, and two invented tox environments (py34/py36) that don’t exist in tox.ini. Pattern-completed claims traceable to “a project like this,” not to the code. The skill arm caught that same 3.4 claim and demoted it to a sourced limitation instead of asserting it.

And the skill kept its own single fabrication in the report rather than hiding it. On csrgenerator, its field table claimed an empty CN value returns HTTP 400. A missing CN is indeed 400 — but an empty one hits the code’s own raise KeyError("CN cannot be empty"), which is unhandled and returns HTTP 500 (verified live). One wrong behavioral detail in a table cell, in a run that otherwise executed pytest (23 passed, exact match) and a real curl CSR generation. The skill isn’t magic; it’s discipline, and discipline has a residual error rate. We logged it.

Where the skill was unambiguously sharper was verified specifics: “pip install -e . installed requests==2.34.2 / PyJWT==2.13.0” matched a fresh venv exactly; the killport limitations (Windows LISTENING-only match, unconditional SIGKILL, one-port-per-invocation) all traced to source with the kill flow verified end to end. Claims that came from a run, not a hunch.

The honest caveats

Our methodology requires the weaknesses printed next to the wins, and there are real ones here.

Single rater, not a 3-developer panel. The preference and trust rows are one expert judgment by the benchmark author, made with the source open. The protocol calls for ≥3 independent developer raters; that panel wasn’t available in this harness. Read those two rows as indicative, not as the multi-rater result the design specifies.

The fabrication gap is narrow by construction. Because both baselines reused accurate repo docs, the baseline had fewer chances to invent. On a repo with no existing documentation the gap would likely widen — but we’re reporting what these three repos showed, which is 2 vs 1, and N=3 is directional only.

Command execution depends on the environment. When the agent’s machine can’t run the project, commands are verified statically against manifests and flagged as unexecuted in the log — weaker than a real run, and we mark it as such.

Install

git clone https://github.com/Skillproofdev/readme-discipline ~/.claude/skills/readme-discipline

Restart Claude Code. It triggers on “write a README”, “document this repo”, “create/rewrite README.md”, and README review or audit requests — and stays out of the way for full docs sites, API reference generation, and changelogs. It joins our discipline series: token-discipline cuts what a task costs, research-discipline cuts what research gets wrong, and this one cuts what your documentation fabricates.

FAQ

How is this different from a README generator like readme-ai? Generators produce structure and offload accuracy to you — their own docs tell you to review the output. This skill inverts that: it reads the code first, greps every falsifiable claim against source, runs every documented command in a clean checkout, and hands you a verification log so you can check its work. Structure is the easy part; the skill spends its effort on truth.

What’s the verification log, exactly? A separate artifact in the reply (not committed) that maps every claimed feature to a file:line in source and marks every command as ran ✓ or not executed — verified against <manifest>, plus anything deliberately left out for lack of evidence. If that log would be empty, the skill skipped its own first three rules. It’s the receipt that lets you trust the prose.

Does the fabrication rule make the README shorter and blander? No — the rules are written to add useful content, not cut it. The mandatory limitations section and the verified-command quickstart are things generic READMEs omit. In the benchmark the skill’s READMEs were more complete (18/18 sections) than the baseline’s, not thinner.

Is one fabrication in three repos good enough? It’s better than the baseline’s two, and it’s honest about the residual — an empty-CN behavior wrong by one HTTP status code, logged rather than buried. If your README backs a decision that’s expensive to get wrong, the verification log tells you exactly which claims to spot-check, which takes minutes instead of re-reading the whole codebase.