Testing Claude Skills for Security Review: What Actually Works

The proposition of an AI that can audit code for security flaws is compelling. It suggests a future where common vulnerabilities are caught before the first commit, and complex attack vectors are surfaced automatically. The reality, as with most things in software, is more nuanced. A tool is only as good as its implementation, and in the rapidly expanding world of AI skills, not all implementations are created equal.

At SkillProof, we don’t just list skills; we test them. We install them, run them against real-world code, and publish the results—pass or fail. Our process is built on the premise that transparency is non-negotiable. Of the 743 skills we’ve benchmarked to date, 508 passed our tests. 204 required non-trivial setup to function correctly. And 31 performed worse than using the base model, meaning you are actively better off not installing them. We are the only directory that publishes these failures.

This article details our findings from applying this methodology to a critical category: Claude skills for security review. We’ll cover which skills successfully identified planted vulnerabilities in our test suite and, just as importantly, explore a case where a popular audit skill caused the model to miss a critical bug that plain Claude would have found on its own.

The Baseline: What Plain Claude Finds

Before evaluating any skill, we must establish a baseline. What can the base model—Claude with no skill installed—accomplish on its own? The answer is not zero. Given a snippet of code and a prompt like “Review this code for security vulnerabilities,” the base model is reasonably effective at spotting common, well-documented anti-patterns. It will reliably flag obvious SQL injection vulnerabilities in string-interpolated queries, identify hardcoded secrets, and question the use of deprecated, insecure functions.

However, its knowledge is general. It lacks the deep, domain-specific context required for a comprehensive claude code security audit in specialized fields. It may not recognize a subtle logic bug in a Cosmos SDK module that leads to an inflationary exploit, or a missing nonReentrant modifier in a Solidity contract, because these patterns are not part of its general training data in the same way that strcpy buffer overflows are.

This limitation is why skills exist: to provide that missing context. But what happens when that context is flawed? In one of our benchmarks, we tasked an audit skill with reviewing a piece of code containing a priority-one logic bug. The skill, which was essentially a long, generic checklist, fixated on low-level issues like variable naming and comment density. It completely missed the architectural flaw.

When we ran the same test with the base model, it correctly identified the P1 bug. The skill, in its attempt to be helpful, induced a form of tunnel vision, preventing the model from performing the holistic analysis it was otherwise capable of. This is not a hypothetical risk; it is a documented finding from our own testing methodology.

The Danger of Checklist Tunnel Vision

A well-designed checklist can be a powerful tool. It ensures consistency and prevents simple mistakes from being overlooked. A poorly designed one, especially when applied to a large language model, can be a liability. The failure we observed is a prime example of this.

The failing skill operated by forcing the model’s analysis into a rigid, predefined structure. It asked the model to answer a series of generic questions: “Are inputs validated?” “Is error handling robust?” “Are there comments?” While these are valid questions, they are insufficient for a comprehensive security review.

The critical vulnerability in our test code wasn’t a simple case of unvalidated input. It was a state-management error that could only be identified by understanding the flow of data across multiple functions. The base model, free from the constraint of the checklist, was able to reason about the code’s behavior and spot the anomaly. The skill-guided model, however, was so focused on ticking the boxes that it never performed that higher-level analysis. It saw the trees, but the skill actively hid the forest.

This highlights a fundamental risk in the emerging ecosystem of ai security review skills. A skill that is merely a wrapper around a generic list of best practices can be actively harmful. It provides a false sense of security while potentially blinding the model to the very classes of bugs it is uniquely suited to find. A proper claude skills security review requires more than a simple list; it requires specialized knowledge.

Vetted Skills That Find Real Vulnerabilities

Fortunately, not all skills fall into this trap. The best security skills provide targeted, domain-specific knowledge that demonstrably improves the base model’s performance. They encode patterns and heuristics for niche ecosystems that the base model would not otherwise have. Here are a few examples from our verified tests.

Cosmos SDK: Cosmos Vulnerability Scanner

The Cosmos ecosystem has a unique architecture with its own set of common pitfalls. To test skills in this domain, we created a synthetic Cosmos SDK rewards module with several intentionally planted bugs. One was a subtle map-iteration bug that could lead to non-deterministic behavior, and another was an unvalidated msg_server payout function that failed to check if a user had sufficient funds to claim a reward.

The base model missed all of them. It lacked the specific context to understand the implications of iterating over a Go map (which is non-deterministic by design) within the context of a state machine, or the standard patterns for validating messages in the Cosmos framework.

The Cosmos Vulnerability Scanner (9.2/10, Pass), however, found them. The skill’s internal documentation includes patterns specific to Cosmos development, which it uses to guide the model’s analysis. It correctly flagged the map iteration as a consensus risk and identified the missing validation in the payout logic, providing a clear explanation and a suggested fix. This is a clear win for a specialized skill.

AI/ML Code: AI/ML Attack Surface

Another area with unique risks is the code that powers AI and machine learning systems. Deserialization attacks via pickle files are a well-known vector. We created a 29-line Python file containing four distinct vulnerabilities: insecure deserialization with torch.load, pickle.load, and numpy.load(allow_pickle=True), plus a subtle f-string formatting bug that could lead to prompt injection.

The AI/ML Attack Surface skill (8.4/10, Pass) was designed for this exact purpose. It uses a battery of grep-like checks to find dangerous function calls. It successfully identified all four planted vulnerabilities. However, in the spirit of our honest verdict policy, we must also report its own flaw: the regular expression it used to detect prompt injection had a false negative for a slightly different formatting pattern. The skill is effective, but not perfect—a crucial distinction.

Smart Contracts: Smart Contract Vulnerability Auditor

Smart contract security is a high-stakes field where a single bug can lead to millions in losses. We tested the Smart Contract Vulnerability Auditor (9.2/10, Setup) against a test vault contract seeded with three classic bugs: a reentrancy vulnerability in the withdraw() function, an unchecked return value from an external call, and a simple access-control error.

The skill, which requires some setup to configure its analysis parameters, successfully identified all three. It correctly explained the danger of the external call preceding the balance update in the withdraw() function, flagged the missing check on the call() return value, and pointed out the function that should have been restricted to the contract owner. This is a task where specialized knowledge of the EVM and Solidity patterns is not just helpful, but essential.

General vs. Domain-Specific Security Skills

These examples illustrate a clear pattern. The most effective security skills are either highly specialized or intelligently structured to avoid the checklist trap. We can categorize them broadly.

Skill Type	Best For	Example	Key Finding
Domain-Specific	Niche ecosystems with unique attack patterns	Cosmos Vulnerability Scanner	Catches bugs the base model cannot know about.
Task-Specific	Common but complex development tasks	API Security	Structures code defensively from the start.
Structured Checklist	Broad code review and user-facing security	Wallet Security Review	Guides analysis without causing tunnel vision.

Task-specific skills like API Security (9.6/10, Pass) offer a different kind of value. Rather than finding bugs in existing code, they help write secure code from the start. We tested it by first writing a naive POST /api/orders endpoint in Python, then re-writing it with the skill’s guidance. The skill prompted for authentication and authorization checks, enforced a strict Pydantic schema with input validation, and added rate limiting and structured logging. It transformed a fragile endpoint into a robust one by guiding the development process.

Well-designed checklists also have their place. The Code Review Checklist (9.6/10, Pass) and Wallet Security Review (9.2/10, Pass) are good examples. Unlike the failing skill, their checklists are not a rigid set of yes/no questions. They are structured prompts that direct the model’s attention to specific areas—concurrency, resource management, cryptographic practices—without preventing it from performing a holistic analysis. They act as a focusing lens, not as blinders.

Integrating AI into a Security Workflow

Based on our tests, it’s clear that using an AI for claude vulnerability scanning is not a fire-and-forget process. It cannot replace a dedicated static analysis tool, a dynamic scanner, or, most importantly, a skilled human reviewer. Its role is that of an exceptionally fast, knowledgeable, but sometimes naive, pair programmer.

To use these tools effectively, integrate them into the development loop, not just at the final review stage. Run a skill like API Security while you are writing the code. Use a domain-specific scanner like the Cosmos Vulnerability Scanner as a pre-commit hook to catch common errors in that ecosystem.

The goal is to augment human intelligence, not replace it. The AI can handle the first pass, catching dozens of low-to-medium severity issues and freeing up human engineers to focus on complex architectural design, business logic flaws, and novel attack vectors. For teams looking to streamline this process, adopting AI tools can be a significant force multiplier, as we’ve explored in the context of DevOps workflows.

A Tested, No-Hype Approach to AI Security

The effectiveness of an AI in security review depends entirely on the quality of the tools you give it. A generic, unverified skill can create a dangerous illusion of security. A vetted, domain-specific skill can provide genuine, measurable value by catching bugs the base model would miss.

This is why independent, transparent testing is critical. Without it, you are simply trusting the marketing copy. The difference between a skill that passes a real-world test and one that fails can be the difference between a secure application and a costly breach.

For teams looking to adopt a set of vetted security tools, we’ve bundled our top-performing security skills, including several mentioned here, into a single pack. You can find the Security Pack in our catalog for $10.

Ultimately, building a secure software ecosystem requires a culture of rigorous verification and honest assessment. For more of our research and findings on this topic, see our main post on Claude skills for security.