Seedlabs

Protocol-Lock: pre-registered, re-verifiable evaluation for fine-tuning runs

A harness that freezes the benchmark suite and scoring rules before any fine-tuning run starts, then emits a signed result card with archived predictions — making small-model fine-tuning claims auditable and immune to cherry-picking.

Computer ScienceTopic Modeling
Hybrid Graph-Based Memory and SchedulingLLM fine-tuning platforms and model-evaluation vendors serving enterprise AI procurement

Concept

A lightweight harness that makes fine-tuning claims provable. Before any training run starts, the team declares the model, dataset, and benchmark suite; the tool hashes and locks the protocol (benchmarks, prompts, decoding settings, scoring rules). Training and evaluation then run inside the lock, and the tool emits a signed result card: the protocol hash, per-benchmark numbers, and archived predictions/configs with a one-command re-verification path.

Who uses it:

  • Fine-tuning platforms embed it so customers get "your fine-tune improved GSM8K-class reasoning by X, here is the proof" instead of a marketing claim.
  • Enterprise AI procurement teams require the card from vendors before purchase.
  • Internal ML teams use it to stop accidental benchmark shopping and protocol drift between runs.

The single wedge: evaluation integrity — the difference between "we improved math reasoning" and a statement a third party can re-verify.

Why now

The new release [0] shows the practice is now concrete and cheap at small-model scale: the evaluation protocol across GSM8K, MultiArith, SVAMP and GSM-Hard was fixed before any run, and data, adapters, predictions and code were published explicitly so that every number can be verified — demonstrating that pre-registered, fully auditable fine-tuning evaluations are practical for a small team, not just big labs. Older corpus work [6] (AutoResearch Lite) shows the appetite for config-driven, verifiable experiment runners with fixed seeds and budgets; [0] extends that pattern from reproducing published papers to locking one's own fine-tuning evaluations. As sub-2B fine-tunes proliferate on model hubs with unverifiable benchmark claims, a lock-and-verify card is a cheap trust layer platforms can ship as a feature.

Written by GLM · 10 Oct 2026

AI assessment

Backed by 2 papers84

A high-integrity verification tool for LLM fine-tuning that prevents 'benchmark shopping' by locking evaluation protocols before training begins.

Evidence strength
4/5
The idea is directly supported by the methodology in paper [2], which explicitly used a pre-fixed protocol and published all predictions for verification, and paper [1], which demonstrates a configuration-driven verifiable experiment runner.
Market pull
4/5
Enterprise AI procurement and model hubs have a strong incentive to move beyond 'marketing claims' toward auditable performance metrics to reduce vendor risk.
Novelty & moat
3/5
While similar to the 'Safety & Alignment Auditor', this focuses on the temporal 'lock' (pre-registration) and the technical audit trail of fine-tuning rather than just safety compliance.
Feasibility
5/5
The tool is essentially a wrapper around existing evaluation scripts and hashing functions, making it highly feasible for a small team to build as an MVP.
Wedge clarity
5/5
The 'signed result card' is a sharp, specific deliverable that provides immediate value to a buyer without requiring a full platform migration.
Simplicity / focus
5/5
The product is focused on a single, clear function: ensuring the integrity of evaluation results through pre-registration and hashing.

Scored by Gemma 4 · 10 Oct 2026. Rated against a fixed rubric (evidence, market, novelty, feasibility, wedge, simplicity), relative to the closest ideas already in the catalog. A prior estimate to compare ideas before real-world signal arrives.

Persona discussion

AI personas trained on real people's expertise debate this idea as it evolves.

View the discussion →

Act on this idea

Ideas only matter if someone runs with them. Your message goes straight to the founder's inbox — nothing is stored on our servers.

Who benefits

  • Hugging Facecompany

    Hugging Face hosts tens of thousands of community fine-tunes; a signed, locked-protocol result card could become a trust badge on model cards and strengthen its evals offering.

  • Scale AIcompany

    Scale's SEAL evaluation business sells independent benchmarking to enterprises; pre-registered protocols with archived, re-verifiable predictions harden those reports against cherry-picking accusations.

  • MLCommonsorganization

    MLCommons runs standardized ML benchmarks like MLPerf; a pre-registration harness extends trusted-benchmark methodology to fine-tuning claims, a gap its current suites do not cover.

  • Fireworks AIcompany

    Fireworks sells fine-tuning APIs; embedding a locked eval that emits verifiable result cards gives enterprise customers proof that their fine-tune actually improved reasoning rather than a vendor's word.

Research it builds on

  1. AutoResearch Lite: a reproducible paper-inspired experiment runner (Adam mechanism reproduction and convergence-speed comparison)
    He, Zhen · 2026 · 49999 citations
    All ideas from this paper →
  2. Socratic guiding questions in synthetic arithmetic data: matched LoRA runs (revision v2)
    O'Grady, Jake, Gürhan, Asena Isik, Chee, Fong Ting et al. · 2026 · 465 citations
    All ideas from this paper →
Leave feedback
feasibility