Add LLM Evals
Playbook for wiring reference-based + LLM-as-judge evals into CI, with judge-calibration steps and named anti-patterns.
Getestet · Funktioniert
Was es kann
A methodology skill (no bundled scripts) for adding evaluation - not just tracing - to an LLM/agent app: pick reference-based vs LLM-as-judge per metric, curate a 20-100 example dataset, wire threshold-gated checks into CI via promptfoo/DeepEval/Ragas, then optionally add production LLM-as-judge scoring. Triggers on 'add evals', 'test my prompt', 'is my RAG accurate', or setting up an eval suite in CI; explicitly distinguishes itself from tracing/observability asks.
Testbericht
On 'help me add evals to my RAG chatbot for CI', the baseline free-hand answer was four vague bullet points ('write some tests', 'consider using an LLM to grade'); the skill run named the RAG-specific Ragas quartet, gave a runnable promptfoo config with a CI-gating exit code, and added the judge-calibration step (hand-label ~30 examples, check agreement) the baseline never mentioned.
Getestet am: 2026-07-15 · Claude Code 2.x (agent harness)
Installation
git clone https://github.com/ContextJet-ai/awesome-llm-observability mkdir -p ~/.claude/skills cp -r awesome-llm-observability/skills/add-llm-evals ~/.claude/skills/add-llm-evals
Befehle & Beispiel-Prompts
/add-llm-evalsPlaybook for wiring reference-based + LLM-as-judge evals into CI, with judge-calibration steps and named anti-patterns.
Skills reagieren auf normale Anfragen — keine Slash-Befehle nötig. Nach der Installation aktivieren Prompts wie diese den Skill (auf Englisch):
Help me add evals to my RAG chatbot to catch regressions in CIIs my LLM app actually accurate? I want to test it before shippingHow do I catch prompt regressions before they hit production?