A/B Test Designer
Promises A/B test statistics; body is a 230-word generic template
Test report
- Verdict
- Tested · Didn't pass
- Tested
- Jul 30, 2026
- Environment
- Claude Code 2.x (agent harness)
- Upstream re-checked
- Jul 30, 2026 · 74ec411
⚠ This skill is no longer available upstream. Our re-check on Aug 10, 2026 couldn't find it any more (repo unreachable/deleted). The test below is what we measured on Jul 30, 2026 and we're leaving it up as a record — but there is nothing left to install, so we've removed the command.
Gave it a real experiment - 603/12,050 versus 683/11,980 - and asked whether it was significant and what sample size a 10% lift would need. Without the skill the answer came out complete: z = 2.400, p = 0.0164, 95% CI [+0.13pp, +1.27pp], about 31,207 users per arm for 80% power, plus SRM, peeking and winner's-curse caveats. With the skill the numbers were identical, because the body supplies no test, no formula and no alpha or power default - its actual instructions are 'understand the full context', 'apply best practices' and 'validate inputs before processing'. All 183 SKILL.md files in this repo share that same 230-word template, emitted by a generate_skills.py script still pointing at the author's Windows desktop. The words 'p-value', 'power', 'confidence' and 'Bonferroni' appear nowhere in the file.
Scored on four weighted criteria — install, triggering, output vs. baseline, docs. How scoring works
- Installs cleanly 5/5
- Triggers reliably 5/5
- Output vs. baseline 3/10
- Docs & honesty 1/5
What A/B Test Designer does
Advertised as a skill for designing and analysing A/B tests - sample sizes, significance testing, multiple comparisons and results interpretation. In practice the SKILL.md body contains no statistical procedure, formula, or default, only boilerplate shared verbatim with the other 182 skills in the same repository. Installs cleanly but adds nothing to an A/B test question.
How to install A/B Test Designer
Nothing to install: the source repository no longer has this skill. If the author brings it back, our daily re-check will pick it up and the command will reappear here.
Commands — how to trigger A/B Test Designer
-
/ab-test-designerPromises A/B test statistics; body is a 230-word generic template
It also activates on plain-language prompts like these:
-
Is my A/B test result significant? Control 603/12050, variant 683/11980 -
How many users per variant do I need for 80% power at a 10% lift? -
We ran 5 variants, how do I correct the p-values for multiple comparisons?
Frequently asked questions
- Is the A/B Test Designer skill free?
- Yes. The skill itself is free from inbharatai/claude-skills. SkillProof publishes the install command and an independent test verdict at no cost.
- Does A/B Test Designer work with Claude Code?
- We tested it with Claude Code 2.x (agent harness) on Jul 30, 2026. Verdict: Tested · Didn't pass. Gave it a real experiment - 603/12,050 versus 683/11,980 - and asked whether it was significant and what sample size a 10% lift would need. Without the skill the answer came out complete: z = 2.400, p = 0.0164, 95% CI [+0.13pp, +1.27pp], about 31,207 users per arm for 80% power, plus SRM, peeking and winner's-curse caveats. With the skill the numbers were identical, because the body supplies no test, no formula and no alpha or power default - its actual instructions are 'understand the full context', 'apply best practices' and 'validate inputs before processing'. All 183 SKILL.md files in this repo share that same 230-word template, emitted by a generate_skills.py script still pointing at the author's Windows desktop. The words 'p-value', 'power', 'confidence' and 'Bonferroni' appear nowhere in the file.
- How do I install A/B Test Designer?
- Copy the install command from this page, run it in your terminal, and restart Claude Code. Skills live in ~/.claude/skills/ (global) or .claude/skills/ inside a project.
- Can I use A/B Test Designer with Cursor, Copilot, Gemini CLI, Codex or other AI tools?
- The SKILL.md format is native to Claude (Claude Code, Desktop, claude.ai). The instructions inside adapt to other assistants: Cursor rules, GitHub Copilot instructions, Windsurf rules, Custom GPTs, AGENTS.md for OpenAI Codex, and GEMINI.md for Google Gemini CLI — our conversion guides cover each, and the free converter on the tools page does the wrapping for you.