Catch data leakage in AI-built ML models
Stops vibe-coded ML models from leaking: audit the data first, check every feature exists at prediction time, split by time, beat the market, and report calibrated results with their uncertainty.
Install just this skill
From your terminal, in one command:
claude plugin marketplace add iampreetdave-max/agent-skills && claude plugin install honest-model-eval@iampreetdave-skillsOr from inside Claude Code:
/plugin marketplace add iampreetdave-max/agent-skills/plugin install honest-model-eval@iampreetdave-skillsOr without the plugin system (macOS, Linux or Git Bash):
git clone --depth 1 https://github.com/iampreetdave-max/agent-skills /tmp/agent-skills && mkdir -p ~/.claude/skills && cp -r /tmp/agent-skills/skills/honest-model-eval ~/.claude/skills/Restart Claude Code afterwards. On Windows PowerShell 5, use ; instead of &&. Remove it with claude plugin uninstall honest-model-eval@iampreetdave-skills.
Use it
It loads by itself when what you ask matches, for example:
- “Build a model to predict match winners from this CSV”
- “Is my model leaking?”
- “Why does it score 92% offline and fail live?”
Or call it directly with /honest-model-eval:honest-model-eval. Installed from a pack, the pack name comes first, for example /all-skills:honest-model-eval.
What it does
- Runs a data audit script: size, balance, duplicates, ID columns and leak suspects.
- Writes a column dictionary marked VERIFIED or ASSUMED, and asks before trusting the assumed ones.
- Checks that every feature really exists at the moment of prediction, pre-match models especially.
- Splits by time (walk-forward) and scores the baselines, including the market, before the model.
- Reports calibration, confidence intervals, and what was dropped for leakage.
Needs
Python with pandas. Also in all-skills, senior-pack. Read the full instructions in SKILL.md on GitHub.