How to Test Claude Skills: The SkillProof Protocol

The whole site started with a skill that did nothing. Late 2025, a productivity skill was making the rounds on X with a few thousand stars behind it. We installed it, restarted Claude Code, typed the exact use case from the README, and watched Claude answer as if the skill didn’t exist. Checked the directory: files present, frontmatter valid. Typed a different phrasing. Nothing. The skill never fired once in forty minutes, and nothing on the repo page would have predicted that. Stars measure whether a README is exciting. They say nothing about whether the folder underneath it works.

That evening left a question we couldn’t shake: if a skill with that much attention can be dead on arrival, what does the rest of the ecosystem look like? So we started installing skills on a clean machine and writing down what happened. The answer, documented in our failure data, is that roughly half of community skills fail before they help anyone. This post is the other side of that finding: the exact protocol behind every verdict on SkillProof, in enough detail that you can run it on your own skill before you ship it.

The protocol, step by step

A full test takes between 45 minutes and several days, depending on the category. A document skill proves itself in one sitting; a weekly-review skill has to survive an actual week. Either way the steps are the same, and the order matters, because each gates the next. There is no point scoring output on a skill that never triggers.

Step 0: a fresh environment

Every test starts on a machine profile with an empty skills directory and default Claude Code settings. This sounds like ceremony until the first time it saves you. Skills interact: one can appear to work because another skill on the machine is quietly doing the heavy lifting. We learned this by testing a proposal skill on a machine that already had the docx skill installed. The output looked great. On a clean profile, half the value disappeared, because the document skill had been producing the polished .docx all along.

Step 1: install from the author’s own instructions

We open the repo’s README and follow it literally. Not “we figure out how to install it.” We do exactly what the author wrote, typos and all, because that is what every real user will do. If the commands actually produce ~/.claude/skills/name/name/SKILL.md, one directory too deep, that’s a failed install, even though anyone who knows the format could fix it in ten seconds. We could fix it too. The point is that the newcomer following along at 11pm can’t, and they’ll conclude Claude skills are broken rather than that one path was wrong.

Anything the README doesn’t mention counts against it: undeclared dependencies, instructions written for a Claude Code version two releases back. We note the time from clone to working skill, and whether we had to leave the README to get there.

Step 2: the trigger battery

An installed skill that never activates is decoration. So before any real task we run a five-prompt battery: three phrasings that should fire the skill, two that shouldn’t.

The three positive prompts are deliberately varied. For a Word-document skill: “turn these notes into a report I can send as a .docx,” then “draft this contract as a Word file,” then something oblique like “I need this formatted properly for legal review.” The first is the README’s own example. The second uses different vocabulary for the same intent. The third never names the file format, which tests whether the description covers the job rather than the keywords.

The two negative prompts probe overtriggering, the failure nobody talks about. A document skill that fires when you ask “summarize this document” (pasted text, no file involved) is injecting instructions into conversations where they don’t belong, and you pay for that in context and in weird outputs. A skill that fires on everything is worse than one that fires on nothing; at least the dead one is easy to diagnose.

Five for five is a clean result. Below that, we note which phrasings failed; the pattern is usually diagnostic. A skill that only fires on the README’s exact wording has a description written as a tagline instead of a trigger spec.

Step 3: the baseline run

This is the heart of the test and the reason the protocol exists. We take a real task from the skill’s domain and run it twice: once with the skill installed, once on the naked model with the identical prompt. Then we compare.

The comparison is the only question that matters: is the with-skill output clearly better than what Claude produces anyway? Claude is already good at a lot of things. A “writing improvement” skill competing against a model that writes well has to demonstrate a delta, and most can’t. When we tested frontend-design, we ran the same landing-page brief both ways. The with-skill version had a real type scale and an intentional palette; the baseline had the neon-gradient look everyone recognizes. That delta earned a 10. When the two outputs are hard to tell apart, the skill has no reason to exist, however pleasant its README.

Real inputs matter as much as the comparison. A spreadsheet with malformed headers, an invoice folder where a third of the files are scans. Demo-data performance is marketing; we test the Tuesday-afternoon version of the job, because that’s the version you’ll hand it.

Step 4: the docs cross-check

Last, we read the whole SKILL.md and everything it references, then compare the claims against what we observed. Does the README promise capabilities the skill doesn’t have? Does it disclose its dependencies? Is there anything buried mid-file that looks less like task guidance and more like prompt injection, or a network call the docs never mention?

This step flags maybe one skill in ten, but the ones it catches matter most. A skill is text injected into your model’s context. Reading every line before trusting it is the minimum, and we treat that read as part of the product.

The four scores, and why output counts double

Every tested skill gets four numbers, described on our methodology page. The short version, with what separates a 5 from a 2:

Installs cleanly (out of 5). A 5 means a newcomer following the README gets a working skill on a fresh setup with no detours. A 2 means we eventually got it working through knowledge the README doesn’t contain: fixing paths, reading the source. The skill may be excellent; the door to it is broken.

Triggers reliably (out of 5). A 5 is five for five on the battery: all three positive phrasings fire, both negatives stay quiet. A 2 fires only on wording lifted from its own README, or fires on unrelated work, or both. Common cause in either direction: a description field written to impress humans instead of to inform the model.

Output vs. baseline (out of 10). A 9 or 10 means the with-skill result is unmistakably better on a real task, the kind of difference you’d notice without a scorecard. A 4 means we had to squint. A 2 means the baseline run was as good or better, which happens more than authors would like to believe.

Docs and honesty (out of 5). A 5 means the README matches reality: accurate claims, declared dependencies, nothing undisclosed. A 2 means promises the skill can’t keep or behavior the docs never mention.

Output is scored out of 10 while everything else is out of 5, and that weighting is deliberate: output quality is worth as much as the other criteria combined. Install problems have workarounds. Trigger problems can be patched by editing one description field. But a skill whose output doesn’t beat the baseline is unfixable in any way that matters. The other scores measure whether you can reach the value. The output score measures whether there is any.

Two tests from the log: a 24 and a 17

Numbers mean more with the tests attached. Here is one from each end of the published range.

Systematic-debugging, from Jesse Vincent’s Superpowers collection, scored 24 of 25: install 5, trigger 5, output 9, docs 5. The install is two plugin commands that worked exactly as written, and the trigger battery went five for five. The baseline run is the part we still bring up in conversation: we gave it a race condition that baseline Claude had already “fixed” three times, each fix a guess that moved the symptom around. With the skill loaded, Claude stopped guessing. It formed a hypothesis, wrote a test to check it, watched the test fail, and walked that loop until it found the actual root cause. The docs promise a disciplined debugging process and that is precisely what we watched happen. It’s been a fixture of our coding page since.

Proposal-builder, a community skill, scored 17: install 3, trigger 4, output 7, docs 3. The first run was a failure in the plain sense. Out of the box, on a clean profile, it couldn’t deliver what its README promises: the polished .docx output silently depends on having the docx skill installed, and the branded formatting depends on a proposal template the README barely mentions. Follow the instructions literally, as a new user would, and you get a wall of markdown where a proposal should be. Once we installed the companion skill and set up a template, it assembled a genuinely useful branded proposal from call notes and pricing. The capability is real. The path to it is not in the README, and the scores say exactly that, down to the missing steps spelled out in the test notes.

That gap is the one the star count can’t see. Both repos look competent from the outside. One works the moment you follow its own instructions. The other works only if you already know what it forgot to tell you.

What a verdict means

The scores roll up into one of three verdicts. The middle one confuses people, so let’s be precise.

Pass means the skill installed from the author’s own instructions, triggered correctly, and beat the no-skill baseline on a real task. Of the 73 skills in the catalog, 35 carry this verdict.

Works with setup means the skill delivers real value, but not out of the box. It needs a companion skill or a configuration step first, and the listing says which. Ten skills sit here, and the verdict is not a euphemism for failure. Some skills require setup by design: a brand-guidelines skill is supposed to be useless until you fill in your palette and voice, and a pipeline-review skill can’t review a pipeline it can’t see. The verdict exists so you know what an honest half hour of configuration buys before you spend it.

In test queue means we listed the skill because it looks promising and haven’t finished testing it. No verdict is implied in either direction; 28 skills are waiting. Skills that fail testing outright don’t get a quiet deletion, either: the test notes say what we ran and what broke, because a documented failure is more useful to you than a gap in the catalog.

Retesting, because Claude keeps changing

A verdict is a snapshot, and the ground under it moves. Skills sit on top of a model, and models get updated. A description that triggered reliably in one Claude Code release can start misfiring in the next, because triggering depends on how the model reads descriptions, and that reading shifts. Drift is not hypothetical; we’ve watched a reliable skill start ignoring one of its three positive phrasings after a release, with not a character of the skill changed.

So every listing carries a tested date and the Claude Code version, and major releases put the whole pass list back in the retest queue, most-installed skills first. When a verdict changes, the listing changes. An aging tested date is your cue to weigh the verdict accordingly; it’s the honest cost of testing against a moving platform.

What we don’t test, and where the method is weak

A protocol you can’t criticize is a protocol nobody described honestly. The known limits:

Sample tasks can’t cover every use. We run one or two real tasks per skill, chosen to be representative, and a skill that shines on our 40,000-row spreadsheet may still stumble on your 400,000-row one. The verdict is evidence, never a guarantee.

The docs score leans on one tester’s judgment. Reading a SKILL.md for honesty is closer to editing than to measurement, and two careful readers can weigh the same vague sentence differently. We publish test notes partly so you can audit us.

We don’t test at scale or over long horizons. One clean machine, days rather than months. Slow degradation and workflows involving several skills at once are outside the method’s reach for now.

Security review is a read, not an audit. We check for undisclosed network calls and injection-shaped instructions, but a determined bad actor could get something past a manual read. Treat our docs score as a filter, and keep your guard up for anything that touches credentials.

And the baseline itself moves. “Beats naked Claude” means naked Claude on the test date; as the base model improves, some passing skills will see their delta shrink toward zero. One more reason retesting isn’t optional.

Running the protocol on your own skill

If you’re about to publish a skill, a condensed version of this takes about an hour and puts you ahead of half the ecosystem.

1. Fresh profile. Empty skills directory, default settings. Your daily machine hides your bugs.
2. Install from your README only. Better: hand the README to someone who’s never seen the repo and watch. Every question they ask is a missing sentence.
3. Run the five-prompt battery. Three phrasings that should fire, including one that never uses your keywords, plus two adjacent prompts that shouldn’t. Fix misses by rewriting the description field, not by adding README caveats.
4. Do the baseline comparison. Same task, with and without your skill. If you can’t tell the outputs apart, rethink what the skill is for before you publish.
5. Reread your SKILL.md as a skeptic. Every claim you can’t demonstrate, cut. Every dependency, declare.
6. Lint the format. Frontmatter mistakes are the most preventable failure class we see, and a validator catches them in seconds.

FAQ

How long does it take to test a Claude skill?

The condensed self-test takes about an hour. Our full protocol runs 45 minutes for a simple document skill and up to a week for skills whose value only shows over time, like weekly-review skills. The trigger battery takes minutes; the baseline comparison is where the hours go.

Can I test a skill without a second machine?

Yes. You need a clean profile, not clean hardware. Point Claude Code at an empty skills directory (or move yours aside) and you get the isolation that matters: no other skills competing for triggers, none quietly covering for the one under test.

What’s the most common reason skills fail testing?

Output that doesn’t beat the baseline, at roughly 35% of failures, with broken installs close behind at 30%. The install failures sting most because they’re the cheapest to prevent: the author never followed their own README on a machine that wasn’t theirs.

How do I get my skill tested and listed on SkillProof?

Submit it here with the repo link. It enters the discovery queue, gets triaged by traction and category fit, then goes through the protocol on this page. Run the self-test first and your odds of a pass verdict go up, because you’ll catch the same defects we would.

The protocol isn’t clever. It’s a clean machine, a README taken at its word, five prompts, one honest comparison. What makes it work is that nobody else in the pipeline does even that much: authors test on their own machines, and stars measure excitement. The gap between those two is where that dead productivity skill from late 2025 lived. We keep closing it, one install at a time.