Walk-Forward Leakage Auditor
Audituje zpětné testy časových řad na únik dat, než důvěřujete Sharpeho poměru
Otestováno · Funguje
Co umí
Strukturovaný auditní kontrolní seznam pro časově založené hodnocení strojového učení: walk-forward backtests, rolling retraining, forecasting pipelines and any chronological train/test split. Nutí agenta uvést prediction-time contract, trace data from raw inputs through features, labels, splits and fitted transforms, then work named red-flag lists for split logic, feature timing, transform scope, label construction, simulation realism and experiment comparability. Spouští se, když požádáte o revizi nebo změnu backtest, rolling-CV or forecasting code.
Testovací report
Provedl jsem audit head to head on a 140-line walk-forward equity backtest built on a seeded random walk, so the honest answer was known in advance: there is no signal in the data. The unaudited script reported out-of-sample AUC 0.636 and Sharpe 2.00 and concluded ship it; a normal no-checklist review caught the forward-return column in the feature matrix, the full-sample z-score and the scaler fitted before the split, which brought it to AUC 0.531 and Sharpe 1.05 — still a ship decision. Working the SKILL.md checklist section by section additionally caught the missing five-day embargo between train and test, a merge_asof with direction=forward on a macro series published three weeks after its period end, an entry threshold tuned on the test Sharpe, and a benchmark computed over a different date range and capital base; after those fixes AUC fell to 0.511 and the strategy lost to buy-and-hold on the matched window, flipping the go/no-go conclusion. A failable guard written per the skill's step 4 fails on both the original and the baseline-fixed file and passes only on the fully audited one. The repo's own worked examples reproduce exactly — 428.6 vs 873.7 MAE, matching the numbers printed in EXAMPLE.md.
Testováno: 2026-07-30 · Claude Code 2.x (agent harness)
Instalace
git clone https://github.com/mospira/walkforward-audit.git mkdir -p ~/.claude/skills/walk-forward-leakage-auditor cd walkforward-audit && cp SKILL.md ~/.claude/skills/walk-forward-leakage-auditor/SKILL.md
Příkazy a ukázkové prompty
/walk-forward-leakage-auditorAudituje zpětné testy časových řad na únik dat, než důvěřujete Sharpeho poměru
Skilly se spouštějí běžnými požadavky — žádné příkazy k zapamatování. Po instalaci ho aktivují prompty jako tyto (anglicky):
Review my walk-forward backtest before I trust this Sharpe ratioMy forecasting model scores 0.95 AUC in backtest but fails live, what's wrongI'm adding rolling retraining to this pipeline, check the split logic