Walk-Forward Leakage Auditor
Prüft Zeitreihen-Backtests auf Datenlecks, bevor Sie dem Sharpe-Verhältnis vertrauen
Getestet · Funktioniert
Was es kann
Eine structured audit checklist für time-based machine learning evaluations: walk-forward backtests, rolling retraining, forecasting pipelines und any chronological train/test split. Sie lässt den agent den prediction-time contract angeben, trace data from raw inputs through features, labels, splits and fitted transforms, then work named red-flag lists for split logic, feature timing, transform scope, label construction, simulation realism and experiment comparability. Wird ausgelöst, wenn Sie ask to review or change backtest, rolling-CV or forecasting code.
Testbericht
Das audit head to head auf einem 140-line walk-forward equity backtest durchgeführt, built on a seeded random walk, so the honest answer war known in advance: there ist no signal in the data. Das unaudited script reported out-of-sample AUC 0.636 and Sharpe 2.00 and concluded ship it; a normal no-checklist review caught the forward-return column in the feature matrix, the full-sample z-score and the scaler fitted before the split, which brought it to AUC 0.531 and Sharpe 1.05 — still a ship decision. Working the SKILL.md checklist section by section additionally caught the missing five-day embargo between train and test, a merge_asof with direction=forward on a macro series published three weeks after its period end, an entry threshold tuned on the test Sharpe, and a benchmark computed over a different date range and capital base; after those fixes AUC fell to 0.511 and the strategy lost to buy-and-hold on the matched window, flipping the go/no-go conclusion. A failable guard written per the skill's step 4 fails on both the original and the baseline-fixed file and passes only on the fully audited one. Die repo's own worked examples reproduce exactly — 428.6 vs 873.7 MAE, matching the numbers printed in EXAMPLE.md.
Getestet am: 2026-07-30 · Claude Code 2.x (agent harness)
Installation
git clone https://github.com/mospira/walkforward-audit.git mkdir -p ~/.claude/skills/walk-forward-leakage-auditor cd walkforward-audit && cp SKILL.md ~/.claude/skills/walk-forward-leakage-auditor/SKILL.md
Befehle & Beispiel-Prompts
/walk-forward-leakage-auditorPrüft Zeitreihen-Backtests auf Datenlecks, bevor Sie dem Sharpe-Verhältnis vertrauen
Skills reagieren auf normale Anfragen — keine Slash-Befehle nötig. Nach der Installation aktivieren Prompts wie diese den Skill (auf Englisch):
Review my walk-forward backtest before I trust this Sharpe ratioMy forecasting model scores 0.95 AUC in backtest but fails live, what's wrongI'm adding rolling retraining to this pipeline, check the split logic