The Karpathy-Inspired CLAUDE.md: A Reality Check on Adversarial Planning Skills

A GitHub repository with a single CLAUDE.md file has accumulated over 100,000 stars as of this writing. It’s often referred to as the “Karpathy skill,” a reference to Andrej Karpathy’s work in AI and education. This popularity raises a critical question for any developer trying to get more out of large language models: does this pattern actually work?

Before we analyze the results from our tests, two points of clarification are essential. First, the repository in question was created by Forrest Chang; it is Karpathy-inspired, not Karpathy-authored. Second, the file is a CLAUDE.md, not a SKILL.md. It’s a set of instructions for a human to copy and paste into a chat session, not a formal skill that can be installed to modify model behavior programmatically.

At SkillProof, we don’t test copy-paste prompts. We test skills. We install them, run them against real-world code, and measure their performance against the base model. To investigate the hype, we identified and tested a cohort of skills that implement the same underlying pattern as Chang’s CLAUDE.md: adversarial planning. This article presents our findings.

Deconstructing the “Grill-Me” Pattern

The core idea behind the popular CLAUDE.md is a form of structured self-critique. The prompt instructs the model to not just produce an answer, but to adopt a persona—a panel of expert critics—to challenge and refine its own output before presenting a final version. The process typically looks like this:

1. Initial Plan: The model generates a high-level plan to solve the user’s request.
2. Self-Critique: The model is prompted to “grill” its own plan, identifying potential flaws, edge cases, and implementation risks. It might list weaknesses or ask clarifying questions of itself.
3. Refined Output: Based on the critique, the model produces a more robust, final answer.

This technique is a form of adversarial planning. You’re forcing the model to act as its own red team, simulating a review process that would normally require a second person or a separate verification step. The hypothesis is that this internal dialogue produces a more considered and correct result, especially for complex tasks like system design or algorithm implementation.

This is the pattern we set out to test. When developers ask for a karpathy claude skill review, this is the mechanism they’re asking about. Does forcing a model to critique itself lead to measurably better code?

CLAUDE.md vs. SKILL.md: A Critical Distinction

The difference between a CLAUDE.md and a SKILL.md is not just semantic; it’s fundamental to how we evaluate performance.

A CLAUDE.md is a manual prompt. Its effectiveness can vary wildly depending on the user’s ability to adapt it, the specifics of their input, and the state of their chat session. It’s a recipe, not a tool. You can’t benchmark a recipe in a standardized way.

A SKILL.md, as defined and used in the SkillProof directory, is a canonical, version-controlled file that programmatically alters the model’s system prompt. When you use a skill from our directory, it’s installed once. Every subsequent request to the model benefits from (or is hindered by) that skill’s instructions, without any manual copy-pasting. This allows for repeatable, objective testing.

Our entire testing methodology is built on this principle. We take a SKILL.md, install it, and run it against a battery of real-world tasks. We compare its output—in terms of correctness, efficiency, and adherence to requirements—against the exact same model with no skill installed. The resulting score is a direct measure of the value added (or subtracted) by that skill.

For this investigation, we did not test Forrest Chang’s file directly. Instead, we sourced multiple skills from the community that formalize the adversarial planning pattern into a reusable SKILL.md format. This allowed us to answer the question: does the pattern itself, when applied consistently, deliver on its promise?

Our Test Methodology for Adversarial Skills

To conduct a fair grill-me claude skill tested analysis, we selected a representative set of tasks that are common pain points for developers and where a more “thoughtful” model could theoretically excel:

• Complex Refactoring: Rewriting a monolithic function with high cyclomatic complexity into smaller, testable units.
• API Client Generation: Writing a client library for a moderately complex OpenAPI specification, including error handling and request/response models.
• Algorithm Implementation: Implementing a non-trivial algorithm from a prose description, such as A* pathfinding or a priority queue.
• Unit Test Generation: Writing a comprehensive suite of unit tests for a class with multiple dependencies and edge cases.

For each task, we ran two trials: one with plain Claude (the base model, no skill) and one with an adversarial planning skill installed. We scored the outputs based on a rubric that includes functional correctness, code quality, completeness, and efficiency. The final score for a skill represents its average performance delta across all tested tasks.

This rigorous process is how we’ve evaluated all 1416 skills currently tracked in our system. It’s a noisy landscape: only 889 (63%) of those skills pass our bar for providing a net positive benefit. Another 467 require non-trivial setup or are only useful in very specific contexts. The adversarial planning skills we tested fell into all of these categories.

The Verdict: Does the Karpathy-Inspired Skill Actually Work?

The answer is nuanced. The effectiveness of the adversarial planning pattern is highly dependent on the complexity of the task. It is not a universal improvement. For some tasks, it’s actively detrimental.

Our tests showed a clear trend:

Task Type	Plain Claude Performance	Adversarial Skill Performance	Verdict
Simple Boilerplate (e.g., a React component)	Fast, 95% correct	Slower, over-critiques, 90% correct	Negative Impact
Complex Refactoring	Often misses edge cases	Catches more edge cases, but verbose	Net Positive
Algorithm Design from scratch	Prone to logical gaps	Better logical structure, slower	Net Positive
Debugging Obscure Errors	Often suggests surface-level fixes	Explores deeper root causes	Net Positive

For simple, well-defined tasks, the adversarial pattern adds unnecessary overhead. The model spends tokens and time critiquing a plan that was already sufficient. In some cases, the self-critique process even introduced errors, as the model hallucinated flaws and then “fixed” them, breaking perfectly good code. This is a crucial finding for anyone wondering if the karpathy claude md worth it for daily coding.

However, for complex, open-ended tasks—the kind that often stump a junior developer—the pattern provides a measurable benefit. When asked to design a system or refactor a tangled piece of legacy code, the self-critique phase forces the model to consider interactions and edge cases that plain Claude often misses. The final output is more robust and requires less human correction, even if it takes longer to generate and is significantly more verbose.

One adversarial-planning skill we tested clearly beat the baseline on open-ended system-design tasks, yet scored below it on plain boilerplate generation. This highlights the need for applying the right skill to the right job, rather than seeking a single “god prompt” to rule them all.

The Hidden Cost: Verbosity and Negative Scores

The most immediate drawback of this pattern is verbosity. A response from a skill using adversarial planning can be 3-5x longer than a response from plain Claude. It includes the plan, the full critique, and then the final answer. While the intermediate steps can offer insight into the model’s “thinking,” they also increase token consumption and the cognitive load on the developer who has to read it all.

More concerning is the risk of negative performance. A poorly implemented skill is worse than no skill at all. At SkillProof, our most important finding is not the list of skills that work, but the list of those that don’t. As of today, 60 skills we have tested scored BELOW plain Claude. They actively make the model less accurate, less efficient, or less reliable.

Several of the adversarial planning skills we tested fell into this category. The failure mode was consistent: the critique phase would get stuck in a loop, or the “expert personas” would contradict each other, leading to a confused and incorrect final output. In one test, a skill aimed at SQL query optimization entered a critique loop where it debated the merits of JOIN vs. INNER JOIN (which are functionally identical in most dialects) and failed to produce any query at all.

This is the anti-hype reality of the AI skill ecosystem. Popularity and GitHub stars are not correlated with performance. A clever prompt pattern can be just as likely to harm as it is to help. The only way to know is to test it.

So, Is the Pattern Worth It?

Let’s return to the original question. Is the pattern from the Karpathy-inspired CLAUDE.md worth it?

As a learning tool, absolutely. Reading through Chang’s CLAUDE.md and similar prompts is an excellent way to understand the concept of chain-of-thought and self-correction. Experimenting with it manually can help you develop better prompting intuition.

As a production tool in the form of an installed SKILL.md, the answer is a firm “it depends.” Our data shows that for specific, high-complexity tasks, a well-implemented adversarial skill can be a powerful tool for senior engineers. It can act as a tireless, logical sounding board for complex problems. For day-to-day coding, it’s likely to be slow, expensive, and potentially counter-productive.

This is precisely the problem SkillProof was built to solve. Instead of relying on hype or star counts, you can rely on our data. We separate the skills that provide a real, measurable lift from the ones that are just clever but ineffective prompts.

Related reading: the share of skills that actually beat plain Claude · how we run every skill before listing it.

We’ve tested dozens of skills that use adversarial planning and other advanced techniques. To see which ones passed our real-world tests and earned a SkillProof score, you can browse the Productivity & Workflow category in our directory. We also bundle the top-performing skills across all categories into a starter pack for $10, giving you a set of verified tools that actually work.