← All skills
honest-model-eval · Claude Code skill

Catch data leakage in AI-built ML models

Stops vibe-coded ML models from leaking: audit the data first, check every feature exists at prediction time, split by time, beat the market, and report calibrated results with their uncertainty.

Install just this skill

From your terminal, in one command:

claude plugin marketplace add iampreetdave-max/agent-skills && claude plugin install honest-model-eval@iampreetdave-skills

Or from inside Claude Code:

/plugin marketplace add iampreetdave-max/agent-skills
/plugin install honest-model-eval@iampreetdave-skills

Or without the plugin system (macOS, Linux or Git Bash):

git clone --depth 1 https://github.com/iampreetdave-max/agent-skills /tmp/agent-skills && mkdir -p ~/.claude/skills && cp -r /tmp/agent-skills/skills/honest-model-eval ~/.claude/skills/

Restart Claude Code afterwards. On Windows PowerShell 5, use ; instead of &&. Remove it with claude plugin uninstall honest-model-eval@iampreetdave-skills.

Use it

It loads by itself when what you ask matches, for example:

Or call it directly with /honest-model-eval:honest-model-eval. Installed from a pack, the pack name comes first, for example /all-skills:honest-model-eval.

What it does

  1. Runs a data audit script: size, balance, duplicates, ID columns and leak suspects.
  2. Writes a column dictionary marked VERIFIED or ASSUMED, and asks before trusting the assumed ones.
  3. Checks that every feature really exists at the moment of prediction, pre-match models especially.
  4. Splits by time (walk-forward) and scores the baselines, including the market, before the model.
  5. Reports calibration, confidence intervals, and what was dropped for leakage.

Needs

Python with pandas. Also in all-skills, senior-pack. Read the full instructions in SKILL.md on GitHub.