Add LLM Evals
Playbook for wiring reference-based + LLM-as-judge evals into CI, with judge-calibration steps and named anti-patterns.
Test report
- Verdict
- Tested · Works
- Score
- Tested
- Jul 15, 2026
- Environment
- Claude Code 2.x (agent harness)
- Upstream re-checked
- Aug 10, 2026 · 782061d
On 'help me add evals to my RAG chatbot for CI', the baseline free-hand answer was four vague bullet points ('write some tests', 'consider using an LLM to grade'); the skill run named the RAG-specific Ragas quartet, gave a runnable promptfoo config with a CI-gating exit code, and added the judge-calibration step (hand-label ~30 examples, check agreement) the baseline never mentioned.
Scored on four weighted criteria — install, triggering, output vs. baseline, docs. How scoring works
- Installs cleanly 5/5
- Triggers reliably 5/5
- Output vs. baseline 9/10
- Docs & honesty 5/5
What Add LLM Evals does
A methodology skill (no bundled scripts) for adding evaluation - not just tracing - to an LLM/agent app: pick reference-based vs LLM-as-judge per metric, curate a 20-100 example dataset, wire threshold-gated checks into CI via promptfoo/DeepEval/Ragas, then optionally add production LLM-as-judge scoring. Triggers on 'add evals', 'test my prompt', 'is my RAG accurate', or setting up an eval suite in CI; explicitly distinguishes itself from tracing/observability asks.
How to install Add LLM Evals
git clone https://github.com/ContextJet-ai/awesome-llm-observability
mkdir -p ~/.claude/skills
cp -r awesome-llm-observability/skills/add-llm-evals ~/.claude/skills/add-llm-evals
Skills live in ~/.claude/skills/ (global) or .claude/skills/
(per-project). Restart Claude Code after installing.
Commands — how to trigger Add LLM Evals
-
/add-llm-evalsPlaybook for wiring reference-based + LLM-as-judge evals into CI, with judge-calibration steps and named anti-patterns.
It also activates on plain-language prompts like these:
-
Help me add evals to my RAG chatbot to catch regressions in CI -
Is my LLM app actually accurate? I want to test it before shipping -
How do I catch prompt regressions before they hit production?
Frequently asked questions
- Is the Add LLM Evals skill free?
- Yes. The skill itself is free from ContextJet-ai/awesome-llm-observability. SkillProof publishes the install command and an independent test verdict at no cost.
- Does Add LLM Evals work with Claude Code?
- We tested it with Claude Code 2.x (agent harness) on Jul 15, 2026. Verdict: Tested · Works. On 'help me add evals to my RAG chatbot for CI', the baseline free-hand answer was four vague bullet points ('write some tests', 'consider using an LLM to grade'); the skill run named the RAG-specific Ragas quartet, gave a runnable promptfoo config with a CI-gating exit code, and added the judge-calibration step (hand-label ~30 examples, check agreement) the baseline never mentioned.
- What is the Add LLM Evals SkillProof Score?
- 9.6/10 — installs cleanly 5/5, triggers reliably 5/5, output vs. baseline 9/10, docs & honesty 5/5.
- How do I install Add LLM Evals?
- Copy the install command from this page, run it in your terminal, and restart Claude Code. Skills live in ~/.claude/skills/ (global) or .claude/skills/ inside a project.
- Can I use Add LLM Evals with Cursor, Copilot, Gemini CLI, Codex or other AI tools?
- The SKILL.md format is native to Claude (Claude Code, Desktop, claude.ai). The instructions inside adapt to other assistants: Cursor rules, GitHub Copilot instructions, Windsurf rules, Custom GPTs, AGENTS.md for OpenAI Codex, and GEMINI.md for Google Gemini CLI — our conversion guides cover each, and the free converter on the tools page does the wrapping for you.