Is the Dataset Evaluation skill free?

Yes. The skill itself is free from awslabs/agent-plugins. SkillProof publishes the install command and an independent test verdict at no cost.

What is the Dataset Evaluation SkillProof Score?

9.2/10 — installs cleanly 5/5, triggers reliably 5/5, output vs. baseline 9/10, docs & honesty 4/5.

How do I install Dataset Evaluation?

Copy the install command from this page, run it in your terminal, and restart Claude Code. Skills live in ~/.claude/skills/ (global) or .claude/skills/ inside a project.

Can I use Dataset Evaluation with Cursor, Copilot, Gemini CLI, Codex or other AI tools?

The SKILL.md format is native to Claude (Claude Code, Desktop, claude.ai). The instructions inside adapt to other assistants: Cursor rules, GitHub Copilot instructions, Windsurf rules, Custom GPTs, AGENTS.md for OpenAI Codex, and GEMINI.md for Google Gemini CLI — our conversion guides cover each, and the free converter on the tools page does the wrapping for you.

Dataset Evaluation

Validates JSONL fine-tuning datasets against SageMaker SFT/DPO/RLVR schemas, per-line errors

by awslabs · awslabs/agent-plugins

Tested · Works

Install — copy the command Download source ↗

Test report

Verdict: Tested · Works
Score: 9.2/10
Tested: Jul 21, 2026
Environment: Claude Code 2.x (agent harness)
Upstream re-checked: Aug 10, 2026 · da6c7cf

Installed into a temp HOME (real ~/.claude/skills confirmed untouched) and ran the bundled format_detector.py against a 240-line DPO dataset I generated with 4 buried defects. Baseline (head/tail eyeballing, no skill) concluded "format looks correct, ready to train" and missed all four; the skill run classified the file as open_weights_dpo and flagged line 89 (unterminated-string JSON), 137 (missing 'rejected' -> unknown), 201 ('rejected' null not str), and 226 ('chosen' list not str), exit 1. Frontmatter parses with name+description; all three referenced files return HTTP 200; no security smells. Only friction: boto3 is imported at module top level so it is needed even for local files (present on my box at 1.43.27; a clean venv raised ModuleNotFoundError).

Scored on four weighted criteria — install, triggering, output vs. baseline, docs. How scoring works

Installs cleanly 5/5
Triggers reliably 5/5
Output vs. baseline 9/10
Docs & honesty 4/5

What Dataset Evaluation does

Detects the format of a local or S3 JSONL dataset and validates it against SageMaker fine-tuning schemas (Nova, GPT-OSS, open-weights SFT/DPO, Verl/RLVR, and eval) before a training job. Triggers on phrasings like "is my dataset okay", "check my training data", or "evaluate my data", running a bundled format_detector.py that samples up to 1MB and reports the detected format plus line-numbered errors. Reports whether the data is ready and points to the dataset-transformation skill when it is not.

How to install Dataset Evaluation

git clone --depth 1 https://github.com/awslabs/agent-plugins.git /tmp/dataset-evaluation-src
mkdir -p ~/.claude/skills
cp -R /tmp/dataset-evaluation-src/plugins/sagemaker-ai/skills/dataset-evaluation ~/.claude/skills/dataset-evaluation
# Dependency: scripts/format_detector.py imports boto3 at module top level, so it is
#   required even for local-file validation:  pip install boto3
# Run directly:  python ~/.claude/skills/dataset-evaluation/scripts/format_detector.py <path-or-s3-uri>
#   add --json for machine-readable output; exit code 0 = valid, 1 = invalid/error
# Plugin-marketplace alternative: the skill ships in the sagemaker-ai plugin of the
#   awslabs/agent-plugins marketplace (.claude-plugin/marketplace.json).

Skills live in ~/.claude/skills/ (global) or .claude/skills/ (per-project). Restart Claude Code after installing.

Commands — how to trigger Dataset Evaluation

/dataset-evaluation Validates JSONL fine-tuning datasets against SageMaker SFT/DPO/RLVR schemas, per-line errors

It also activates on plain-language prompts like these:

Check if my dataset is ready for SFT fine-tuning
Validate this DPO dataset schema before training
Evaluate my training data for format issues

Frequently asked questions

Is the Dataset Evaluation skill free?: Yes. The skill itself is free from awslabs/agent-plugins. SkillProof publishes the install command and an independent test verdict at no cost.
Does Dataset Evaluation work with Claude Code?: We tested it with Claude Code 2.x (agent harness) on Jul 21, 2026. Verdict: Tested · Works. Installed into a temp HOME (real ~/.claude/skills confirmed untouched) and ran the bundled format_detector.py against a 240-line DPO dataset I generated with 4 buried defects. Baseline (head/tail eyeballing, no skill) concluded "format looks correct, ready to train" and missed all four; the skill run classified the file as open_weights_dpo and flagged line 89 (unterminated-string JSON), 137 (missing 'rejected' -> unknown), 201 ('rejected' null not str), and 226 ('chosen' list not str), exit 1. Frontmatter parses with name+description; all three referenced files return HTTP 200; no security smells. Only friction: boto3 is imported at module top level so it is needed even for local files (present on my box at 1.43.27; a clean venv raised ModuleNotFoundError).
What is the Dataset Evaluation SkillProof Score?: 9.2/10 — installs cleanly 5/5, triggers reliably 5/5, output vs. baseline 9/10, docs & honesty 4/5.
How do I install Dataset Evaluation?: Copy the install command from this page, run it in your terminal, and restart Claude Code. Skills live in ~/.claude/skills/ (global) or .claude/skills/ inside a project.
Can I use Dataset Evaluation with Cursor, Copilot, Gemini CLI, Codex or other AI tools?: The SKILL.md format is native to Claude (Claude Code, Desktop, claude.ai). The instructions inside adapt to other assistants: Cursor rules, GitHub Copilot instructions, Windsurf rules, Custom GPTs, AGENTS.md for OpenAI Codex, and GEMINI.md for Google Gemini CLI — our conversion guides cover each, and the free converter on the tools page does the wrapping for you.