Why ‘Awesome’ Lists Aren’t Enough: A Data-Driven Look at Claude Skill Reliability

Every developer knows the pattern. You’re exploring a new ecosystem—in this case, Claude skills—and your first stop is a community-curated list, likely a GitHub repository titled ‘awesome-claude-skills’. These lists are valuable for discovery. They aggregate hundreds of tools in one place, giving you a broad overview of what’s possible. But discovery is not validation. A high star count and a well-written README.md are poor proxies for whether a skill will actually work when you try to run it on a real task.

The core issue is that curation is often a measure of popularity, not reliability. A skill gets added to a list because it has an interesting premise or is made by a known developer. It gets starred by people who think the idea is neat. Very few of those stars represent a user who installed the skill, integrated it into a workflow, and confirmed it performs as advertised. This gap between perceived quality and tested reality is where developers lose hours to debugging and frustration. The search for a truly awesome claude skills list reliable enough for production use often ends in disappointment.

This article examines the difference between curated selections and a catalog built on rigorous, independent testing. We’ll look at the data from our own process to show why you can’t trust a list that doesn’t publish its failures.

The Curation Fallacy: Popularity vs. Performance

When we talk about curated claude skills vs tested ones, we’re talking about two fundamentally different models of verification. Curation relies on social proof and surface-level indicators:

• GitHub Stars: A measure of interest, not function.
• Author Reputation: A good developer can still publish a broken or poorly maintained skill.
• README.md Claims: Marketing copy for a tool. It describes the ideal state, not the current, potentially buggy one.
• Date of Last Commit: A useful but incomplete signal. A skill can be recently updated and still fail on complex inputs.

These signals are helpful for filtering out completely abandoned projects, but they tell you nothing about a skill’s actual performance. Does it handle edge cases? Does it require three undocumented environment variables to run? Does it silently fail and return a plausible but incorrect result? Curation doesn’t answer these questions. Testing does.

At SkillProof, we don’t curate. We test. We install every skill in a clean environment and run it against a standardized, real-world task relevant to its purpose. We document the process, record the outcome, and assign a score. Our findings reveal a significant disconnect between the skills people share and the skills that actually work.

A Catalog Built on Failure

Our entire premise is built on a simple, transparent process: we run the code. We publish the results, good or bad. This provides a level of best claude skills list accuracy that is impossible to achieve through curation alone. You can read the full details of our process on our /methodology page, but the top-level statistics paint a clear picture.

As of today, we have installed and tested 1576 distinct Claude skills. Here is the breakdown of the results:

• 992 (63%) passed our tests and received a score of 5/10 or higher. These skills perform their advertised function correctly on our test case.
• 518 required non-trivial, often undocumented, manual setup to even run. We flag these as Needs Setup so developers know what they’re getting into.
• 66 skills scored below the baseline. This is the most critical finding: using these skills produces a worse result than not installing any skill at all and just using plain Claude. A curated list will never tell you this.

That 63% pass rate is the key figure. It means that if you pick a skill at random from a typical unverified list, you have a greater than 1-in-3 chance that it will either fail, require complex setup, or actively make your output worse. This is an unacceptable rate of failure for anyone trying to build reliable applications.

The Anatomy of a Failed ‘Awesome’ Skill

Let’s consider a common example we’ve seen dozens of times. A skill for analyzing and refactoring code is featured prominently on a curated list. It has hundreds of stars. The README.md shows a clean, simple example of it transforming a messy function into an elegant one.

When we tested it, the reality was different:

1. Installation: The requirements.txt file specified a dependency with a version that has been deprecated and conflicts with modern libraries.
2. Execution: Running the skill on our test file—a moderately complex 200-line script—caused it to hang indefinitely. It only worked on the simplistic, 10-line example from its own documentation.
3. Output: When we finally got it to run on a simpler file, the refactored code it produced had syntax errors and failed to pass a basic linter check.

This skill would be a celebrated entry on an ‘awesome’ list. In our tested catalog, it would receive a failing verdict and a detailed run log explaining exactly why it fell short of plain Claude on the task. The table below summarizes the difference in perspective:

Metric	Curated List View	SkillProof Tested Verdict
Signal	GitHub Stars, README.md claims	Pass/Fail on real task, /10 score
Setup	Assumed pip install	Documented setup steps, or Needs Setup flag
Performance	Author’s description	Measured against plain Claude baseline
Failure	Not visible or acknowledged	Published as a failing verdict with a run log

Another skill we tested, designed for interacting with a popular API, passed its core test. However, it required the user to manually create a configuration file in a specific format that was not mentioned anywhere in the SKILL.md or the linked repository. It took 45 minutes of digging through the source code to figure it out. A curated list would just link to it. We flag it as Needs Setup and provide the exact configuration file we used to make it work, saving the next developer 45 minutes.

The Compounding Problem of Unverified Skills

For a developer using a single skill for a one-off task, a 37% chance of failure is an annoyance. For anyone building systems that compose multiple skills, it’s a critical flaw. The reliability of a chain of tools is the product of the reliability of each component.

Imagine you’re building an agent that uses three skills: one to read a file, one to analyze its contents, and one to summarize the findings. If we use our catalog’s average pass rate of 63% as a proxy for the reliability of any randomly chosen skill, the probability of all three succeeding in the chain is:

0.63 * 0.63 * 0.63 = 0.25

A 25% chance of success. This is why a proper composio awesome claude skills review or any analysis of tool-composing systems must start with the verified reliability of the individual components. Without it, you are building on a foundation of sand. Chaining together ‘awesome’ skills that have not been independently tested is an exercise in building complex, brittle systems that are guaranteed to fail.

The only way to build robust, multi-skill agents is to use components that have been verified to work. You need to know the setup requirements, the expected inputs, and the performance baseline for every piece of your stack. A simple link in a markdown file doesn’t provide that information.

How to Vet a Skill Beyond the README

If you find yourself evaluating a skill from an unverified source, you have to become your own tester. This is time-consuming but necessary if you don’t have access to a pre-tested catalog. Here are the steps we recommend, which mirror our own internal process:

1. Isolate and Install: Never install a new skill directly into your primary development environment. Create a clean, virtual environment (venv, conda, etc.) and install it there. Check the dependencies it pulls in. Are they ancient, or do they have known vulnerabilities?
2. Analyze the SKILL.md: Look for more than just a description. Is there a clear schema for arguments? Does it define the tool’s function signature, inputs, and output format? A lack of a clear interface is a major red flag. We discuss this in more detail in our post on what makes a good skill definition.
3. Design a Real-World Test Case: Don’t just use the example provided by the author. Find or create a realistic piece of data or a scenario that represents your actual use case. If it’s a code refactoring skill, give it a messy file from one of your own projects. If it’s a data analysis skill, use a real-world dataset, not a perfect 5-row CSV.
4. Run and Measure: Execute the skill and check the output. Does it work? Is the output correct? How does its performance and quality compare to what you would get by just prompting the base model directly? This baseline comparison is crucial. If the skill doesn’t provide a significant improvement over plain Claude, it’s just adding complexity for no benefit.

This process is effective, but it’s also a significant time investment for every single skill you want to try. The goal of a tested directory is to perform this work once, for the entire community, and make the results public.

Finding Skills That Actually Work

Curated lists are a great starting point for seeing what the community is excited about. But excitement doesn’t run code. For building real applications, you need tools that have been proven to work under realistic conditions. The gap between a star on GitHub and a passing test on a real-world file is where most projects falter.

Our data shows that a significant portion of publicly available skills are, in their current state, broken, difficult to set up, or simply not better than using the base model. Publishing this data isn’t about criticizing developers; it’s about providing the ground truth needed to make informed engineering decisions. The 66 skills we found that perform worse than plain Claude are not ‘bad’ tools, but they are tools that developers should avoid until they are improved. You will not find this warning on an ‘awesome’ list.

Instead of manually vetting every promising tool from a community list, you can use a catalog where that work has already been done. Every skill listed includes its score, a run verdict, and the exact setup we used.

Related reading: For more on why community popularity and real quality diverge, see popular vs. good Claude skills. And to understand the ground truth behind every verdict in our catalog, read how we test Claude skills.

Browse our catalog of over 900 passing skills, sortable by score and category, to find tools you can trust for your next project. Start with the most reliable skills for coding that we’ve tested so far.