Seedlabs

ShadowParam: A Config-Code Hyperparameter Divergence Linter for ML Repos

A tool for developers that finds hidden settings in AI code that don't match the official config file.

A CI tool that flags ML hyperparameters whose declared config value differs from the value hardcoded on the executed code path, and logs the effective value behind every result — catching the shadow-hyperparameter failures that silently invalidate published numbers.

NeuroscienceNeural dynamics and brain function
AI Governance and Cognitive ImpactMLOps / ML research engineering — configuration integrity and reproducibility checking in CI for training and evaluation repositories.

Concept

A developer tool, shipped as a GitHub Action and CLI, that does one job: make hyperparameter provenance trustworthy. Statically, it maps every tunable declared in config files, YAMLs, or CLI defaults to every code site that consumes it, and flags divergences — a constant hardcoded in the training module that shadows the config value, a default that differs from the documented value, a second gate on a different variable that the config never reaches. Dynamically, it records the effective value of each parameter for every run, so each logged metric is traceable to the exact values that produced it, and it flags cross-run comparisons made under differing confounds. The output is a pre-publication (or pre-release) report: these hyperparameters have config-code divergence, and your numbers came from the code path, not your config. The buyer is any ML research or engineering team whose results must be reproducible, from academic labs to enterprise MLOps.

Why now

The source paper's Version-2 correction is a recent, fully documented incident of precisely this failure mode: the gate was specified at 0.5 in config.py while the training module gated a different quantity (the consistency score) at 0.3 — and every published number came from the 0.3 path, a divergence discovered only after publication and requiring a formal correction that also withdrew the headline series-best BWT claim (this paper's 15.19% vs. a prior paper's 16.03%, compared across runs with different AA) and reclassified a supposed falsification test as a mere determinism check. A divergence linter with effective-value run logging would have caught the 0.5/0.3 mismatch and the confounded cross-run comparison before publication. The failure is generic to any ML repository with configs, defaults, and multiple code paths; the fix is a bounded static-analysis plus run-logging capability — a small, sellable tool rather than a platform.

Written by GLM · 9 Oct 2026

AI assessment

Backed by 1 paper93

A highly focused, high-utility developer tool that solves a specific, painful failure mode in ML research (config-code divergence) with a clear technical path to MVP.

Evidence strength
5/5
The idea is directly derived from a detailed, real-world failure case (Version-2 correction) where a specific mismatch between config.py and manas.py invalidated headline results.
Market pull
4/5
High urgency for academic labs and enterprise MLOps teams where reproducibility is a requirement for publication or production deployment.
Novelty & moat
5/5
Unlike the listed 'Verification' or 'Policy' bots, this targets a specific technical divergence between configuration and execution paths, filling a distinct gap in the MLOps toolchain.
Feasibility
4/5
Combining static analysis (AST parsing of configs and code) with dynamic runtime logging is a well-understood engineering task that a small team could prototype quickly.
Wedge clarity
5/5
The wedge is extremely sharp: a CI linter that flags 'Shadow Parameters' before a researcher hits 'submit' on a paper or 'merge' on a PR.
Simplicity / focus
5/5
The product is a single-purpose tool (a linter/logger) rather than an over-scoped platform, maintaining a tight focus on configuration integrity.

Scored by Gemma 4 · 9 Oct 2026. Rated against a fixed rubric (evidence, market, novelty, feasibility, wedge, simplicity), relative to the closest ideas already in the catalog. A prior estimate to compare ideas before real-world signal arrives.

Persona discussion

AI personas trained on real people's expertise debate this idea as it evolves.

View the discussion →

Act on this idea

Ideas only matter if someone runs with them. Your message goes straight to the founder's inbox — nothing is stored on our servers.

Who benefits

  • Weights & Biasescompany

    Their tracking already ingests configs; adding divergence detection and effective-value logging turns their reproducibility story into a hard guarantee for enterprise ML customers, with this public correction incident as a marquee use case.

  • Comet MLcompany

    Experiment-tracking customers routinely hit silent config drift; a linted, provenance-verified run history is a direct differentiator for their platform.

  • Hugging Facecompany

    Hosts thousands of training repositories with configs and defaults; a provenance lint fits their existing config-checking CI culture and improves trust in released models.

  • MLCommonsorganization

    Benchmark integrity (e.g., MLPerf) depends on reported configurations matching executed ones; shadow-hyperparameter detection protects the comparability of submitted results.

  • Allen Institute for AIorganization

    Releases fully open training stacks (e.g., OLMo); config-code provenance linting directly serves their fully reproducible-science mandate.

Research it builds on

  1. Maya-Manas: Oscillatory Thalamo-Cortical Gating for Class-Incremental Learning in Affective Spiking Neural Networks
    Venkatesh Swaminathan · 2026 · 136 citations
    All ideas from this paper →
Leave feedback
feasibility