We Failed 3 Times Before This Humanizer Beat the Baseline

Most humanizer tools are an unmeasured pattern list. The biggest one on the market ships 33 patterns, 28.6k stars, and zero numbers — no detector scores, no meaning check, English only. The paid SaaS crowd publishes numbers, but they’re the wrong ones: claimed 99% bypass rates that independent tests clock at ~66%, achieved by paraphrasers that garble your facts on the way through.

We build tested Claude skills for a living, so we set out to build a humanizer we could actually measure. And measuring it nearly killed it. We published three SKILL.md versions that lost a blind human-ness test to the raw, un-processed AI text. The fourth won. This post is the whole arc — the three losses included — because the losses are the reason you should trust the win.

The bar we set before writing a line

The setup was fixed in advance. Twelve AI-generated texts across genres built to stress different failure modes: blog intros, product copy, client emails, fact-dense technical explainers, a formal memo (register stress), a numbers-heavy earnings summary, one non-English text. Each one gets rewritten twice — once by a baseline Claude agent told simply “rewrite so it doesn’t sound AI-generated, keep the meaning,” and once by an identical agent that reads our SKILL.md first.

Three metrics, all pre-registered:

1. A scripted, reproducible count of mechanical AI tells (of 103 in the corpus).
2. A claim-by-claim meaning audit against ground-truth inventories written before any rewrite (130 claims total).
3. A blinded A/B human-ness preference: which rewrite reads more human?

The ship bar was blunt: the skill has to win, or clearly tie, the blind preference against the naive baseline. A humanizer a blind reader prefers the un-processed version of has no reason to exist.

One honest caveat up front, because it governs everything below. The detector APIs we planned to use (GPTZero, Sapling, ZeroGPT) were all key- or payment-gated when we ran — curl-verified and logged. So “AI-likeness” here is a blinded in-context LLM judge, not a detector score, and we label it that way everywhere. The tell count and the claim audit are the objective, reproducible numbers.

v1: over-corrected into corporate sludge

The first version rewrote aggressively. Swap the tells for richer vocabulary, restructure with a free hand. The result read worse.

Trading “leverage” for a grander synonym doesn’t remove a tell — it re-inflates the text. v1 sanded off the obvious words and replaced them with the register of a press release. Worse, the aggressive rewriting drifted meaning: it dropped two pre-registered claims (an audience-framing sentence in the remote-work blog, a word from a leadership triple in the Russian text) and went corporate-stiff where the source was warm.

Blind preference: base 8, v1 4. The judge kept picking the baseline for keeping a light human framing that v1 flattened. First version, first loss.

v2 and v3: under-corrected, tells left standing

The v1 diagnosis looked obvious: it over-rewrote. So we swung the other way — surgical, minimal edits that protect readability and meaning. That fixed the facts perfectly and lost harder.

v2 nailed meaning preservation: 130/130 claims intact, zero altered, dropped, or added. But minimal editing under-corrects. It left “cornerstone,” “paradigm shift,” “I hope this email finds you well,” “personal health command center” standing on the page. Thirteen residual tells against the baseline’s four. The blind judge flagged exactly those phrases and rated v2 markedly more machine-made.

Blind preference: base 12, v2 0. Every single text.

v3 tried to split the difference — a tell-to-plain-word replacement table plus a zero-tolerance re-scan gate: remove every tell, replace plainer never fancier. It shaved the residual tells from 13 to 12 and changed nothing that mattered.

Blind preference: base 12, v3 0. Again, every single text.

The insight after three losses: it was never about vocabulary

Three versions, zero blind-preference wins — cumulatively 4-0-28 including v1, and 0-0-24 across the two surgical attempts. When you lose that consistently, the problem isn’t a knob you set wrong. It’s the frame.

Here’s what the data was screaming. The baseline wins because it restructures into a human voice: it varies rhythm, opens inside the subject, drops the promotional scaffolding. Every one of our skill versions stayed welded to the source’s AI skeleton — the topic-sentence opener, the uniform paragraph shape, the rule-of-three list, the promo framing. We were polishing vocabulary on a structure that announced “machine” before a single word was read.

That’s the core tension, demonstrated three times over: our unique selling point — freeze the meaning — is exactly what kept the skill hugging the original so tightly it could never out-human a free rewrite. Fact-safety and human-ness were pulling against each other, and structural fidelity lost the reader every time.

AI text gives itself away by its shape, not its word list. Swapping “delve” for “look at” leaves the skeleton standing, and the reader still feels the machine underneath.

v4: freeze the claims, rebuild the structure

The fix was a method inversion, not another tuning pass. Stop protecting the structure. Protect only the claim set — every fact, number, name, date, quote, and caveat, plus the direction and strength of each claim — and rebuild everything else freely. Resequence. Merge sentences. Write new openings. Cut scaffolding. Do what a sharp human does: read the passage, understand what it claims, then say it again in your own structure.

The obvious risk is that free restructuring is the classic way to drift facts — so v4’s claim audit was run extra-strict, and the mandatory final pass walks the original claim by claim against the rewrite, restoring any drift even at the cost of a cleaner line.

It worked. v4 is the first version to beat the baseline: blind preference 8-4. AI-likeness essentially tied (26 vs 27), and the lowest tell count of any arm — 2. The judge’s reasons for the wins were all structural: cold opens that start inside the subject (the earnings summary now leads with the revenue figure like a real wire lede), listy scaffolding merged into prose, register tightened but kept in-genre (the memo stays formal, the email stays professional).

And the facts held. 129 of 130 claims intact — 99.2%, zero altered, zero added, zero caveats lost. Every number, name, date, price, and the one verbatim CEO quote survived. Free restructuring did not break the payload.

The honest 5-way table

Every arm, every number, negative results included:

Metric	Base	v1	v2	v3	v4 (ships)
Mechanical tells left (of 103)	4	4	13	12	2
AI-likeness, median¹ (lower = more human)	27	32.5	46.5	45	26
Claims intact (of 130)	127 (97.7%)	127 (97.7%)	130 (100%)	130 (100%)	129 (99.2%)
altered / dropped / added facts	2 / 1 / 0	1 / 2 / 0	0 / 0 / 0	0 / 0 / 0	0 / 1 / 0
Blind human-ness preference (skill W-T-L vs base)	—	4-0-8	0-0-12	0-0-12	8-0-4

¹ Blinded in-context LLM judge (0–100), same model family as the rewriter — labeled honestly, not a detector. Base’s median wobbled 27–30 between rounds; that’s judge variance, disclosed, not smoothed.

The single v4 miss: the OAuth explainer dropped one acronym gloss — it kept “the PKCE extension” and its purpose but not the full-name expansion “Proof Key for Code Exchange.” Counted strictly as one dropped claim for honesty. It’s an explanatory gloss, not a number, name, date, quote, or caveat, and no claim drifted. That’s the exact failure mode free-restructuring risks, it showed up once in 130 claims, and it’s now patched into the re-read step (“preserve acronym expansions”).

What we do not claim

The win is real, but it is not a blowout, and pretending otherwise would defeat the point of running the benchmark. Several texts were near coin-flips (25 vs 26, 26 vs 27, 26 vs 26). This is in-context single-judge scoring from the rewriter’s own model family — not an external detector — and the corpus was authored by the same project. Treat the 8-4 preference and the AI-likeness numbers as directional: “v4 clears the bar,” not “v4 dominates.” The tell count and the claim audit are the solid metrics.

We do not promise detector results. Detectors disagree on 15–25% of texts and false-positive on real human writing. This skill rebuilds text into a human voice; it does not sell a bypass guarantee, and if your context requires AI disclosure — academic work, publisher policy — disclose it. A humanizer is an editor, not a disguise. A human-rater or live-detector round is the confirmation we recommend before anyone calls the 8-4 final.

Why this makes SkillProof trustworthy

We could have run four versions, buried the three losses, and shipped a page that said “our humanizer beats the baseline 8-4.” Every competitor in this niche effectively does that — asserts a number, shows none of the work behind it.

We published the three failures instead. The v1 over-correction, the two surgical losses at 0-12, the diagnosis, the inversion. You can read all of it, re-run the scripted tell count yourself, and check every claim audit in the repo. That is the whole proposition: we test other people’s skills for a living, and our own skills get the same treatment — measured, with negative results attached.

It joins our discipline series: token-discipline cuts what your prompts cost, research-discipline cuts what your research gets wrong, and this one cuts what your writing gives away.

Install

git clone https://github.com/Skillproofdev/text-humanizer ~/.claude/skills/text-humanizer

Restart Claude Code. It triggers on “humanize this,” “this sounds like AI / like ChatGPT,” “make it sound natural,” and AI-detection worries — in any language. It stays out of the way for writing new content from scratch, translation jobs, and non-text tasks.

FAQ

Why publish the three failures at all? Because they’re the evidence. Anyone can assert “8-4 over baseline.” The three losses show the win wasn’t luck or a tuned judge — it came from a specific, testable insight (structure beats vocabulary) that only surfaced after three honest defeats. The failures are what make the number believable.

Does this beat AI detectors? We can’t say, and we won’t pretend to. The detector APIs we planned to use were all key- or payment-gated at run time, so our “AI-likeness” figure is a blinded in-context LLM judge, not a detector score. The skill removes tells and rebuilds text into a human voice; it does not sell a detector-bypass guarantee. Detectors disagree on 15–25% of texts and false-positive on genuine human writing.

Will restructuring my text change what it says? That’s the whole design constraint. The claim set — every number, name, date, quote, and caveat, plus the direction and strength of each claim — is frozen, and a mandatory final pass walks the original claim by claim against the rewrite to restore any drift. In the benchmark, free restructuring kept 129 of 130 claims intact (0 altered, 0 added), the single miss being a non-load-bearing acronym gloss now patched.

Is the 8-4 margin big enough to trust? It clears our pre-set bar — win the blind preference — where three prior versions lost. But it’s a clear win, not a blowout, from a single in-context judge on a self-authored corpus. Treat it as directional and confirm the load-bearing decisions yourself; the tell count and claim audit are the metrics we’d stake the strongest claim on.