Walk-Forward Leakage Auditor
Auditerer tidsserie-backtests for lækage, før du stoler på Sharpe
Testet · Virker
Hvad det gør
En structured audit checklist for time-based machine learning evaluations: walk-forward backtests, rolling retraining, forecasting pipelines and any chronological train/test split. It makes the agent state the prediction-time contract, trace data from raw inputs through features, labels, splits and fitted transforms, then work named red-flag lists for split logic, feature timing, transform scope, label construction, simulation realism and experiment comparability. Udløses når du beder om at gennemgå eller ændre backtest, rolling-CV or forecasting code.
Testrapport
Ran the audit head to head on a 140-line walk-forward equity backtest built on a seeded random walk, so the honest answer was known in advance: there is no signal in the data. The unaudited script reported out-of-sample AUC 0.636 and Sharpe 2.00 and concluded ship it; a normal no-checklist review caught the forward-return column in the feature matrix, the full-sample z-score and the scaler fitted before the split, which brought it to AUC 0.531 and Sharpe 1.05 — still a ship decision. Working the SKILL.md checklist section by section additionally caught the missing five-day embargo between train and test, a merge_asof with direction=forward on a macro series published three weeks after its period end, an entry threshold tuned on the test Sharpe, and a benchmark computed over a different date range and capital base; after those fixes AUC fell to 0.511 and the strategy lost to buy-and-hold on the matched window, flipping the go/no-go conclusion. A failable guard written per the skill's step 4 fails on both the original and the baseline-fixed file and passes only on the fully audited one. The repo's own worked examples reproduce exactly — 428.6 vs 873.7 MAE, matching the numbers printed in EXAMPLE.md.
Testet: 2026-07-30 · Claude Code 2.x (agent harness)
Installation
git clone https://github.com/mospira/walkforward-audit.git mkdir -p ~/.claude/skills/walk-forward-leakage-auditor cd walkforward-audit && cp SKILL.md ~/.claude/skills/walk-forward-leakage-auditor/SKILL.md
Kommandoer og eksempelprompter
/walk-forward-leakage-auditorAuditerer tidsserie-backtests for lækage, før du stoler på Sharpe
Skills udløses af almindelige forespørgsler — ingen kommandoer at huske. Efter installationen aktiverer prompter som disse skillen (på engelsk):
Review my walk-forward backtest before I trust this Sharpe ratioMy forecasting model scores 0.95 AUC in backtest but fails live, what's wrongI'm adding rolling retraining to this pipeline, check the split logic