Claude Summaries That Don't Quietly Rewrite the Facts

Ask Claude to summarize a document and you get a clean, confident digest. What you can’t see — because you didn’t read the source — is everything the summary quietly changed on the way down: a “may reduce risk” that became “reduces risk”, a $2.4M that became $2.9M, two conflicting dates collapsed into one. A summary is lossy compression of someone else’s claims, and the only legitimate loss is omission. Everything else is distortion the reader has no way to catch.

So we measured it. The result is summary-discipline, our newest skill: nine enforceable faithfulness rules, benchmarked before/after against a frozen ground-truth sheet. It’s free and MIT-licensed: github.com/Skillproofdev/summary-discipline.

The niche gap: 174 summarizers, zero faithfulness rules

Before writing a rule we searched our index of 16,682 discovered skills. 460 mention summarizing; 174 are genuine summarizers. Every single one optimizes the same thing: structure and brevity — sections, bullet counts, action items, tone. Zero state a single checkable rule about the relationship between the summary and its source: nothing added, numbers verbatim, hedges intact, attribution preserved.

That’s the strange part, because academia has been measuring exactly these failures for a decade. FactCC, FRANK, SummaC, AggreFact, FaithBench — a whole literature of summarization-faithfulness benchmarks with named error types. No published skill imports any of it. summary-discipline is that missing transfer: hallucination-benchmark error categories turned into generation-time rules you can audit line by line.

Nine rules, each auditable against the source

The full SKILL.md is nine rules; the load-bearing ones:

1. No claim without a source sentence. If you can’t point at the passage that licenses a sentence, it doesn’t go in. Genuinely needed context gets marked [context, not in source].
2. Numbers are copied, never re-derived. Value, unit, and referent verbatim — no rounding, averaging, or silent arithmetic. “$4.2M Q3 revenue, up 12% YoY”, never “roughly $4M”.
3. Preserve the hedge. “May reduce risk” never becomes “reduces risk”; “confirmed” never softens to “might”. The modality is the last word you’re allowed to cut.
4. Preserve attribution. “The CEO claims margins improved” ≠ “margins improved”. Never merge two speakers into one consensus nobody voiced.
5. Flag contradictions; never resolve them silently. Two conflicting figures stay two figures, in a visible Flagged block. The summary does not pick a winner or average.

Plus a declared compression contract (~4,800 words → ~200 words · importance order), omission rules that protect decisions, deadlines, risks and reversing caveats at any length, quote anchors on load-bearing claims, and a claim-by-claim self-audit before delivery.

The benchmark: every claim audited against a frozen sheet

Here’s the honest scope, stated up front. The corpus is 11 documents, one per stress category. These numbers are from a pre-registered 6-document subset — D02 cardiotide trial, D04 quarterly review, D05 incident postmortem, D07 launch transcript, D09 sleep-research abstracts, D11 planted-contradiction memo — chosen by the coordinator before any runs, one per category. The other five aren’t run yet. Treat this as a directional read, not the full-corpus result.

Pre-registration matters here: before any summary is generated, we record for each document its key-fact list, every hedged claim, every attributed claim, and every number with its referent. A “critical omission” is defined against that frozen sheet — not invented after seeing the output. Then two arms per document: baseline (plain “summarize this”) and skill (same prompt, SKILL.md loaded first), same model, same target length, each output atomized and audited claim by claim.

Metric (6-doc totals)	Baseline	Skill
Added claims	0	0
Distorted numbers	0	0
Hedge-drops (certainty upgrades)	1	0
Attribution losses	0	0
Critical omissions	10	2
Contradictions flagged (of 3 planted)	0	3

The star case: a memo that disagrees with itself

D11 is a project status memo with three contradictions planted between its exec summary and its body: a budget of $2.4M vs $2.9M, two different cutover dates, and 6 vs 8 engineers. This is the exact failure mode the skill was built for.

The baseline silently resolved all three. It stated one budget, one cutover date, one headcount — each as settled fact, never once signalling that the memo contradicted itself. A reader of that summary has no way to know the source disagrees. (It happened to pick the internally-consistent figures — $2.9M matches the “58% spent” math — but that’s luck, not disclosure.)

The skill flagged all three, both values presented, none averaged, in a visible Flagged block. This is the cleanest result in the whole benchmark, and the reason is important: contradiction flagging is a behavior, not a word budget. A 160-word summary could have flagged all three conflicts in one line each. The skill also preserved eight self-limiting caveats the baseline dropped while keeping the headline — a research abstract’s “null result should not be interpreted as evidence of safety”, a postmortem’s alert-noise trade-off — the Rule 6 “keep the reversing caveat” cases.

Where it ties, and the confound we won’t hide

Our methodology requires the losses next to the wins, and this run has real ones.

On three metrics, both arms scored zero. Added claims, distorted numbers, attribution losses — tied at 0/0. At these target lengths the baseline model is already disciplined about not inventing facts, mangling numbers, or stripping speakers. The skill didn’t beat it there because there was nothing to beat. The lone baseline hedge-drop (“moderately stable” → “stable”) is borderline and could reasonably be scored zero, which would tie that metric too. And on D07 — the document built to stress modality — the baseline kept the flagship “target, not a commitment” hedge on its own. The skill’s headline advantage didn’t materialize on the doc designed to test it.

And the omission gains partly ride on longer output. This is the confound, and it’s structural, not cosmetic: the skill’s outputs ran about 1.7× longer than the baseline’s (mean compression 2.6× vs 4.5×), and it exceeded the word target on all six documents. Most of the overage is the Flagged + Omitted apparatus appended after a near-target summary body — but on D04 the skill also ran the body itself long, explicitly citing fidelity, and we scored that as a length miss rather than retro-excusing it. The honest reading: a chunk of the “8 fewer omissions” is the skill spending more words. Where it wins on equal footing is (a) contradiction disclosure, independent of length, and (b) D09, where the baseline was on-length yet still dropped three caveats the skill kept.

Full 11-document runs are needed before generalizing. This subset is favorable to the skill but not a clean sweep.

Install

git clone https://github.com/Skillproofdev/summary-discipline ~/.claude/skills/summary-discipline

Restart Claude Code. It triggers on “summarize”, “tl;dr”, “key points of”, meeting digests, and long-document compression — and stays out of the way for creative rewriting, translation, and tone changes. It joins our discipline series: token-discipline cuts what a task costs, research-discipline cuts what research gets wrong, and this one cuts what a summary quietly changes.

FAQ

Does this make summaries longer? On this subset, yes — about 1.7× longer than the baseline, mostly the Flagged block. That’s the confound we flag above, not a feature. The one length-independent win is contradiction flagging: the skill can surface a conflict in a single line without spending a bigger word budget.

How is a “critical omission” defined so it isn’t scored after the fact? Pre-registration. Before any summary exists, we freeze each document’s key-fact list — decisions, headline numbers with referents, reversing caveats. Auditing happens against that sheet, so “you dropped a critical fact” can’t be invented to favor either arm.

Does it replace a human reading the source? No. It cuts silent distortion and surfaces contradictions the baseline buried, but it’s benchmarked on 6 of 11 documents and ties the baseline on three metrics. If a decision is expensive to get wrong, the skill’s quote anchors make verifying the load-bearing claims take seconds — use them.

Why only six documents? Because every claim in every output was hand-audited against a frozen ground-truth sheet. We prefer a small benchmark where the ground truth is real over a large one scored by an unverified judge. The remaining five documents, and the full per-document scores, are published in the repo verdict.