Add LLM Evals
Playbook for wiring reference-based + LLM-as-judge evals into CI, with judge-calibration steps and named anti-patterns.
Testé · Fonctionne
Ce que fait
A methodology skill (no bundled scripts) for adding evaluation - not just tracing - to an LLM/agent app: pick reference-based vs LLM-as-judge per metric, curate a 20-100 example dataset, wire threshold-gated checks into CI via promptfoo/DeepEval/Ragas, then optionally add production LLM-as-judge scoring. Triggers on 'add evals', 'test my prompt', 'is my RAG accurate', or setting up an eval suite in CI; explicitly distinguishes itself from tracing/observability asks.
Rapport de test
On 'help me add evals to my RAG chatbot for CI', the baseline free-hand answer was four vague bullet points ('write some tests', 'consider using an LLM to grade'); the skill run named the RAG-specific Ragas quartet, gave a runnable promptfoo config with a CI-gating exit code, and added the judge-calibration step (hand-label ~30 examples, check agreement) the baseline never mentioned.
Testé le: 2026-07-15 · Claude Code 2.x (agent harness)
Installation
git clone https://github.com/ContextJet-ai/awesome-llm-observability mkdir -p ~/.claude/skills cp -r awesome-llm-observability/skills/add-llm-evals ~/.claude/skills/add-llm-evals
Commandes et exemples de prompts
/add-llm-evalsPlaybook for wiring reference-based + LLM-as-judge evals into CI, with judge-calibration steps and named anti-patterns.
Les skills se déclenchent sur des demandes en langage courant — aucune commande à retenir. Après installation, des prompts comme ceux-ci l'activent (en anglais) :
Help me add evals to my RAG chatbot to catch regressions in CIIs my LLM app actually accurate? I want to test it before shippingHow do I catch prompt regressions before they hit production?