GitHub Stars vs. Tested Performance: A Poor Correlation for Claude Skills

As developers, we use heuristics to navigate the overwhelming volume of open-source tools. One of the most common is repository popularity. When faced with multiple options, sorting by GitHub star count feels like a rational first step. The assumption is that stars are a proxy for quality, a wisdom-of-the-crowds signal indicating that a project is useful, stable, and maintained. For mature ecosystems like web frameworks or databases, this heuristic often holds. For Claude skills, our data shows it frequently breaks down.

At SkillProof, we don’t have access to private install-count data for skills. Frankly, no one outside the platform providers does, which makes any discussion of claude skill install numbers meaning purely speculative. What we do have is the public star count for every skill we test, and our own tested verdict. After installing and running 1416 skills on standardized tasks, we can state with confidence that there is a weak and often misleading correlation between a skill’s star count and its actual, tested performance.

This article examines that disconnect. We will look at why popular skills often fail to deliver and why some of the best-performing skills are found in the long tail of obscurity. The central question is not just whether popular Claude skills are good, but whether popularity itself is a useful metric for quality in this ecosystem. Our findings suggest it is not.

The Allure of Social Proof

It’s easy to understand why stars are the default metric for discovery. A high star count suggests a project has captured the attention of many other developers. This social proof implies a few things: the concept is valuable, the code has been vetted by many eyes, and a community exists to support it. In theory, more users lead to more bug reports, more pull requests, and a more robust tool over time.

This logic underpins most software ecosystems. However, the Claude skill ecosystem has unique characteristics that undermine this model. The barrier to entry is low, leading to a proliferation of skills that are experimental, incomplete, or wrappers around a single prompt. The pace of change in the underlying models is rapid, meaning a skill that worked six months ago may be broken or, worse, suboptimal today due to dependency rot or changes in the base model’s behavior.

Furthermore, stars can be a lagging indicator of quality, or an indicator of hype rather than utility. A clever README.md or a viral post on social media can generate thousands of stars for a project that is little more than a concept. The stars remain long after the initial excitement has faded and the repository lies dormant. This is the reality we encounter daily.

What 1416 Tested Skills Reveal

Our process is straightforward: we find a skill, install it, and run it against a real-world task defined in our testing methodology. The skill either passes, requires manual setup beyond its documented instructions, or it fails. A failure can mean it produces an error, times out, or—most critically—delivers a result that is measurably worse than using plain Claude for the same task.

Here is the top-level breakdown of our findings from 1416 skills tested to date:

• 889 Pass (63%): The skill installs and performs its advertised function correctly on our test case.
• 467 Need Setup (33%): The skill fails out of the box but can be made to work with non-trivial effort, such as manual dependency installation, code modification, or undocumented configuration.
• 60 Fail (4%): The skill is broken, or its output is inferior to the base model. We classify these as a net negative; you are better off not installing them.

The most important takeaway is that over a third of the skills in our catalog do not work as advertised upon installation. This includes a significant number of repositories with high star counts. The simple act of sorting by popularity on a platform like GitHub will inevitably surface skills that are abandonware, require expert-level setup, or are simply broken.

The Anatomy of a High-Star Failure

While we don’t name specific skills in this context, the patterns of failure among popular repositories are consistent. These are not edge cases; they are recurring archetypes of the disconnect between popularity and performance.

One common archetype is the Over-marketed Concept. We’ve tested several skills with thousands of stars that promise to revolutionize a workflow, such as frontend design. When we run our test, the skill produces syntactically invalid code, uses deprecated patterns, or generates a design that is less coherent than what a simple, well-phrased prompt to the base model produces. The high star count reflects excitement about the idea of the skill, not the quality of its execution. This is a key factor when considering frontend-design skill install count quality—the perceived popularity doesn’t guarantee a working product.

Another is the Dormant Giant. This was a well-built, genuinely useful skill at the time of its creation. It gained a large following and many stars. Then, the maintainer moved on. Two years later, its dependencies are out of date, it calls APIs that no longer exist, and it breaks on the current version of the Claude platform. The stars remain, acting as a trap for new users who assume the project is still active and reliable.

Perhaps the most concerning category is the skill that Actively Hinders Performance. We have tested 60 skills that scored below the baseline performance of plain Claude. For example, a skill intended to help with code refactoring might apply rigid, outdated linting rules that make the code less readable, or a data analysis skill might hallucinate API calls for libraries it doesn’t have access to. These skills don’t just fail to help; they actively make the output worse. Many of these underperforming skills have hundreds or even thousands of stars.

In the Long Tail: Finding Low-Profile Winners

Conversely, some of the most effective and reliable skills in our directory have fewer than 50 stars. These are often targeted tools built by developers to solve a specific, personal problem. They do one thing and do it exceptionally well.

These hidden gems don’t have the marketing push of their more popular counterparts. Their README.md might be sparse, and they may not have a slick logo. What they do have is clean, functional code that has been refined through practical use. We found a skill with just a handful of stars that perfectly automates the process of converting complex JSON objects into clear Markdown tables, scoring a 9/10 in our tests. Another, a niche tool for generating database migration scripts, passed our tests flawlessly while larger, more popular tools struggled with different SQL dialects.

These successes highlight the core problem with using popularity as a filter: it optimizes for visibility, not utility. The most visible projects are not always the most valuable. The real value often lies in the long tail of specialized tools, but discovering them requires a systematic, evidence-based approach—not a simple sort-by-stars.

From Social Proof to Ground Truth: A Better Metric

If stars are an unreliable proxy, what is the alternative? The only true measure of a skill’s quality is its performance on a real task. This is the ground truth. The challenge is that establishing this truth for even a single skill requires time and effort: cloning the repo, creating a test environment, crafting a test case, and running the skill.

This is the work we do at SkillProof. Our /10 score isn’t a measure of our opinion. It’s a record of a tested outcome. A high score means the skill passed a repeatable, objective test. A low score means it failed.

Here is how the two metrics compare in practice:

Metric	What It Suggests	What It Often Means in Reality
High Star Count	”This is a high-quality, trusted skill."	"This was popular at some point; may or may not work now.”
SkillProof Score > 7/10	”This skill is likely to work for you."	"We installed and ran this on a real task, and it passed.”
SkillProof Verdict: Fail	”This skill has a bug."	"This skill performed worse than plain Claude on our test.”

When you are deciding whether to install a skill, the question you should ask is not “Is this popular?” but “Does this work?” The 60 skills that scored below the base model are a stark reminder that popularity can be actively misleading.

A Practical Framework for Skill Evaluation

Given the unreliability of popularity metrics, developers need a more robust framework for evaluating Claude skills. Relying on a directory that has already done the testing is the most efficient path, but if you are evaluating a skill on your own, a healthy dose of skepticism is your best tool.

First, treat the star count as a historical artifact, not a current endorsement. It indicates past interest, not present-day quality. Dig deeper.

Second, check the repository’s activity. Look at the date of the last commit. Are there recent, meaningful changes, or was the last update two years ago? Read the open issues. Are users reporting critical failures? Is the maintainer responding? A vibrant issue tracker with active discussion is a much better sign of health than a high star count on a silent repository.

Third, read the source code if you can. Many skills are quite small. You can often get a sense of the code quality and the approach being taken in a few minutes. Look at the SKILL.md file. Does the prompt engineering seem sophisticated, or is it a simple template that you could easily replicate yourself?

Ultimately, the only way to be certain is to test the skill yourself on a non-critical task. This process—clone, install, configure, test, evaluate—is the foundation of reliable evaluation. It is also a significant investment of time, especially when repeated across dozens of potential skills.

Related reading: how many indexed skills actually run · the skills that earned a top verdict.

We built SkillProof because we believe this verification step is essential, and we know that most developers don’t have the time to do it themselves for every tool they consider. We’ve run the tests on 1416 skills so you don’t have to. You can browse all 889 passing skills in our catalog to find tools that are verified to work, or purchase our curated pack of the top 10 tested, general-purpose skills for a one-time price of $10.