Beyond the Hype: We Ran 1416 Claude Skills to Find the Real Number That Work

You have likely seen the numbers: headlines and directories advertising over 23,000 Claude skills available for installation. This figure suggests a vast, mature ecosystem of tools ready to augment the base model’s capabilities. It’s an impressive number, but it raises a critical technical question: what does “available” actually mean? In most cases, it means a SKILL.md manifest was found in a public code repository. It’s a count based on file discovery, not functional verification.

This approach is simple, scalable, and ultimately misleading. It tells you nothing about whether a skill will install, run without errors, or perform its advertised function effectively. It doesn’t tell you if it’s an abandoned project, a broken proof-of-concept, or even if it performs worse than using no skill at all.

At SkillProof, we take a different approach. We don’t count repositories; we install and execute skills against a standardized set of real-world tasks. This article presents our findings from testing 1416 skills. It’s a direct, data-backed answer to the question of how many Claude code skills there are that actually work, and an examination of whether Claude skill marketplaces are reliable sources for production-ready tools.

The Flaw in Counting Files

The fundamental problem with a headcount of 23,000+ skills is that it treats discovery as validation. Scraping platforms like GitHub for SKILL.md files is a trivial task. The resulting number makes for good marketing copy, but it’s a vanity metric that ignores the reality of software development.

A repository containing a skill manifest is just a starting point. It’s a claim, not a guarantee. When we began our process of systematically testing skills, we quickly identified common failure patterns that simple file-counting misses entirely:

• Incomplete or Malformed Manifests: The SKILL.md file exists, but it’s missing required sections, points to non-existent tool definitions, or is syntactically incorrect. The skill is impossible to install without manual correction.
• Broken Dependencies: The skill’s code relies on external libraries that are outdated, have breaking changes, or are no longer available. The installation might succeed, but the skill will fail at runtime.
• Undocumented Environment Requirements: A skill might require specific environment variables, a running local service, or authentication tokens that are not mentioned in its documentation. One skill we tested required a specific version of a database running on localhost:5433, a detail discovered only by reading its Python source code. Without this, it was non-functional.
• Abandoned Projects: The repository hasn’t been updated in years. The code was written for a previous version of the Claude API and is no longer compatible.
• “Prompt-as-a-Skill”: Some skills contain no actual tools. They are simply elaborate prompts packaged in a skill format. While potentially useful, they don’t represent a functional extension of the model’s capabilities and often perform no better than a well-written prompt.

Simply counting these repositories inflates the perceived size and health of the ecosystem. It creates a landscape where finding a working, reliable tool is a matter of trial and error. Our testing methodology was designed specifically to cut through this noise by making execution the primary measure of a skill’s validity.

Our Findings: A Sobering Look at the Real Count

We installed and attempted to run 1416 skills sourced from various public directories and repositories. Each skill was subjected to a series of automated tests designed to invoke its core functionality. The results provide a much clearer picture of the state of the ecosystem.

Of the 1416 skills we tested, only 889 (63%) passed our initial execution tests without any manual intervention.

Here is a complete breakdown of our findings:

Status	Count	Percentage of Total
Pass (Runs out-of-the-box)	889	63%
Needs Setup (Requires manual config)	467	33%
Fails (Performs worse than base model)	60	4%
Total Tested	1416	100%

Let’s analyze what each of these categories means for a developer trying to use these tools.

Pass (63%): These skills installed correctly and ran on our testbed without errors. This is the “claude skills directory real count” from our sample—the subset of skills that are immediately usable. This is the baseline for what a user should expect from any skill listed in a directory. However, as we’ll discuss, “running” does not automatically mean “high quality.”

Needs Setup (33%): This is a significant and often overlooked category. These 467 skills were not broken, but they were not plug-and-play. Common reasons included:

• Requiring API keys to be manually set as environment variables.
• Needing to connect to a user-provided database or third-party service.
• Depending on local files or system configurations that were not specified in the SKILL.md.

For example, a skill for interacting with a project management API is useless without an API key and an endpoint URL. These skills are not failures, but listing them without clear, upfront setup instructions is a disservice to the user. A directory that doesn’t differentiate between a “Pass” skill and a “Needs Setup” skill presents an unreliable picture.

Fails (4%): This is the most concerning category. These 60 skills not only failed to perform their function but produced results that were actively worse than using the base model with no skill installed. This happens when a skill’s logic is flawed, causing the model to:

• Get stuck in a loop, repeatedly trying to call a broken tool.
• Hallucinate the use of tools that don’t exist in its own definition.
• Misinterpret the user’s intent and apply a tool incorrectly, leading to errors or nonsensical output.

One skill we tested was designed to format code snippets. When given a simple Python function, it attempted to call a format_javascript tool, failed, and returned an error message. The same request to plain Claude would have resulted in a correctly formatted Python snippet. These 60 skills are not just useless; they are detrimental. No reputable directory should list them without a clear warning. We publish these failures because they are a critical part of the data.

Beyond Execution: What Defines Skill Quality?

The data shows that roughly two out of every three skills you might find will run. But this only answers the first part of the question. The second, more important part is about quality. The “23000 claude skills quality” is not a question of quantity, but of performance.

A skill that runs but performs its task poorly is little better than one that doesn’t run at all. This is why, after a skill passes our initial execution test, we score it on a 1-to-10 scale based on its performance on a battery of real-world tasks. Our full scoring rubric is detailed in our methodology, but it centers on a few key principles:

• Reliability: Does the skill consistently succeed at its stated task? Does it use the correct tools for the job?
• Accuracy: Is the output correct and free of errors? If it interacts with an API, does it handle data correctly?
• Efficiency: Does it solve the problem without unnecessary steps or tool calls?
• Graceful Failure: When it encounters an edge case or invalid input, does it return a helpful error message or does it crash?

The difference between a high-scoring skill and a low-scoring one is stark.

A high-scoring skill, like a well-built cloud infrastructure tool, will correctly interpret a request like “list all EC2 instances in us-east-1,” use its list_instances tool with the correct region parameter, handle the paginated response from the API, and present the user with a clean, accurate list.

A low-scoring skill might have the same goal but fail in execution. For instance, a different cloud tool we tested would ignore the specified region and list instances from its default region. It “worked” in that it didn’t crash, but it produced the wrong answer, making it unreliable.

The 60 skills that scored below the base model represent the floor. They are a tangible demonstration that a poorly designed skill is worse than no skill at all. This is a critical data point that is lost when directories prioritize catalog size over verified performance.

Finding the Signal in the Noise

The disparity between the advertised 23,000+ skills and our tested pass rate of 63% highlights the core problem: the ecosystem is full of noise. The true number of functional, high-quality skills is a small fraction of the advertised total.

Manually testing each skill you find is not a practical solution for any developer. The process is time-consuming and resource-intensive. Our test of 1416 skills required significant engineering effort to build the test harness and substantial compute resources to run the evaluations. This is precisely why most directories don’t do it. It’s far easier to run a scraper and publish a large, unverified number.

The goal of a skills directory should be to filter the signal from the noise. It should do the validation work on behalf of the user. This means:

1. Executing every skill: A skill isn’t “verified” until it has been run.
2. Testing for correctness: The skill must be evaluated against real tasks to see if it performs as advertised.
3. Publishing failures: A directory that doesn’t show you what failed is hiding half the story. The failure data is just as important as the success data.

By testing skills across dozens of categories, from data analysis to web development, we build a map of what works, what needs tweaking, and what to avoid entirely.

This data-driven approach is the only reliable way to answer the question, “are claude skill marketplaces reliable?” The answer is: they are only as reliable as their verification process. A marketplace that is just a list of repositories is not a reliable source for professional tools. A directory that runs, tests, and scores every single entry provides a foundation of trust.

Related reading: how many actually beat the baseline · our test methodology.

We’ve done this work across our entire catalog. The 889 skills that passed our tests are available to browse, complete with their scores and a verdict on their performance. For developers who need a core set of proven tools, we offer a curated pack of our highest-rated skills for $10, all guaranteed to run and perform as expected. This is our solution to the signal-to-noise problem: a small, verified, high-quality subset of the massive, unverified public ecosystem.