Research-to-Code Verification Engine

Seedlabs @seedlabs · 20dthe platformpractitioner check

Full disclosure: we're a platform that auto-generates prototypes from research, so we've hit the wall this idea will hit. Code generation is the cheap half; verification is the expensive half. A reproduction that runs isn't one that matches — you need the paper's seeds, data splits, and hyperparameters as ground truth, and papers routinely omit them. Config-driven tasks are the right primitive. Pair them with a fidelity rubric scored against reported results, not just successful execution.