Skill Eval Harness
Python CLI that measures a skill's causal lift: paired with/without runs, leakage lint, ablations
Test report
- Verdict
- Works with setup
- Score
- Tested
- Jul 20, 2026
- Environment
- Claude Code 2.x (agent harness)
- Upstream re-checked
- Aug 10, 2026 · 188ecae
Fetched the recursive tree, README (v0.6.0), pyproject.toml, and the only real SKILL.md (examples/demo-skill/skills/demo/SKILL.md) plus its references/checklist.md — both exist and parse; deps are just pyyaml, MIT-licensed, no exfiltration/hidden-instruction/base64 payloads found. The product is a CLI installed via `uv tool install skill-eval-harness`, so it's inert as a copy-in skill and I can't measure its output without installing an external tool (docked 1 install pt). Trigger judged on the demo-reviewer fixture — SHOULD fire: "Review this diff and label how serious each finding is", "Inspect my proposed code change and mark which findings are blocking", "Go through this PR and triage the issues by severity"; should NOT fire: "Write a new pagination feature for the API", "Summarize this blog post" — I'd make all 5 calls correctly, but the harness itself has no autonomous trigger (it's CLI-invoked), hence 4.
Scored on four weighted criteria — install, triggering, output vs. baseline, docs. How scoring works
- Installs cleanly 4/5
- Triggers reliably 4/5
- Output vs. baseline 3/10
- Docs & honesty 5/5
What Skill Eval Harness does
A repo-agnostic Python CLI (`skill-benchmark`) that measures the causal lift of an Agent Skill by running the same case with and without the skill, grading outputs deterministically, and reporting paired lift, leakage, ablations, trigger rates, and cost. It is a developer tool you invoke from a skill repo's evals/ manifest, not an auto-activating Claude skill. Reach for it when you want to prove a skill actually improves outputs rather than eyeballing it.
How to install Skill Eval Harness
uv tool install skill-eval-harness # external Python CLI (Python 3.10+, uv) — NOT installed here per policy
Skills live in ~/.claude/skills/ (global) or .claude/skills/
(per-project). Restart Claude Code after installing.
Commands — how to trigger Skill Eval Harness
-
/skill-eval-harnessPython CLI that measures a skill's causal lift: paired with/without runs, leakage lint, ablations
It also activates on plain-language prompts like these:
-
Evaluate this skill against a paired baseline variant -
Run trace artifacts through the skill eval harness -
Compare two skill variants to see which one performs better
Frequently asked questions
- Is the Skill Eval Harness skill free?
- Yes. The skill itself is free from adewale/skill-eval-harness. SkillProof publishes the install command and an independent test verdict at no cost.
- Does Skill Eval Harness work with Claude Code?
- We tested it with Claude Code 2.x (agent harness) on Jul 20, 2026. Verdict: Works with setup. Fetched the recursive tree, README (v0.6.0), pyproject.toml, and the only real SKILL.md (examples/demo-skill/skills/demo/SKILL.md) plus its references/checklist.md — both exist and parse; deps are just pyyaml, MIT-licensed, no exfiltration/hidden-instruction/base64 payloads found. The product is a CLI installed via `uv tool install skill-eval-harness`, so it's inert as a copy-in skill and I can't measure its output without installing an external tool (docked 1 install pt). Trigger judged on the demo-reviewer fixture — SHOULD fire: "Review this diff and label how serious each finding is", "Inspect my proposed code change and mark which findings are blocking", "Go through this PR and triage the issues by severity"; should NOT fire: "Write a new pagination feature for the API", "Summarize this blog post" — I'd make all 5 calls correctly, but the harness itself has no autonomous trigger (it's CLI-invoked), hence 4.
- What is the Skill Eval Harness SkillProof Score?
- 6.4/10 — installs cleanly 4/5, triggers reliably 4/5, output vs. baseline 3/10, docs & honesty 5/5.
- How do I install Skill Eval Harness?
- Copy the install command from this page, run it in your terminal, and restart Claude Code. Skills live in ~/.claude/skills/ (global) or .claude/skills/ inside a project.
- Can I use Skill Eval Harness with Cursor, Copilot, Gemini CLI, Codex or other AI tools?
- The SKILL.md format is native to Claude (Claude Code, Desktop, claude.ai). The instructions inside adapt to other assistants: Cursor rules, GitHub Copilot instructions, Windsurf rules, Custom GPTs, AGENTS.md for OpenAI Codex, and GEMINI.md for Google Gemini CLI — our conversion guides cover each, and the free converter on the tools page does the wrapping for you.