Walk-Forward Leakage Auditor
Verifica i backtest di serie temporali per leakage prima di fidarsi dello Sharpe
Promosso
Cosa fa
Una structured audit checklist per le time-based machine learning evaluations: walk-forward backtests, rolling retraining, forecasting pipelines e any chronological train/test split. Fa sì che l'agente dichiari il prediction-time contract, tracci i data from raw inputs through features, labels, splits and fitted transforms, quindi lavori named red-flag lists for split logic, feature timing, transform scope, label construction, simulation realism e experiment comparability. Si attiva quando chiedi di review o change backtest, rolling-CV o forecasting code.
Rapporto di test
Ho eseguito l'audit head to head on a 140-line walk-forward equity backtest built on a seeded random walk, so the honest answer was known in advance: there is no signal in the data. The unaudited script reported out-of-sample AUC 0.636 and Sharpe 2.00 and concluded ship it; a normal no-checklist review caught the forward-return column in the feature matrix, the full-sample z-score and the scaler fitted before the split, which brought it to AUC 0.531 and Sharpe 1.05 — still a ship decision. Working the SKILL.md checklist section by section additionally caught the missing five-day embargo between train and test, a merge_asof with direction=forward on a macro series published three weeks after its period end, an entry threshold tuned on the test Sharpe, and a benchmark computed over a different date range and capital base; after those fixes AUC fell to 0.511 and the strategy lost to buy-and-hold on the matched window, flipping the go/no-go conclusion. A failable guard written per the skill's step 4 fails on both the original and the baseline-fixed file and passes only on the fully audited one. The repo's own worked examples reproduce exactly — 428.6 vs 873.7 MAE, matching the numbers printed in EXAMPLE.md.
Testato il: 2026-07-30 · Claude Code 2.x (agent harness)
Installazione
git clone https://github.com/mospira/walkforward-audit.git mkdir -p ~/.claude/skills/walk-forward-leakage-auditor cd walkforward-audit && cp SKILL.md ~/.claude/skills/walk-forward-leakage-auditor/SKILL.md
Comandi e prompt di esempio
/walk-forward-leakage-auditorVerifica i backtest di serie temporali per leakage prima di fidarsi dello Sharpe
Gli skill si attivano con richieste in linguaggio naturale, senza comandi da ricordare. Dopo l'installazione, prompt come questi lo attivano (in inglese):
Review my walk-forward backtest before I trust this Sharpe ratioMy forecasting model scores 0.95 AUC in backtest but fails live, what's wrongI'm adding rolling retraining to this pipeline, check the split logic