Seedlabs

Frontier Pilot: Measured AI-Impact Trials on a Firm's Own Tasks

A short trial for companies to see exactly which tasks AI improves and which it ruins.

A fixed-scope, ~6-week engagement that runs a small randomized trial of AI on a professional-services firm's own tasks and staff, delivering a measured task-by-task map of where AI boosts speed and quality and where it degrades correctness, plus a per-task AI-use policy.

Computer ScienceArtificial Intelligence Applications
Cross-Corpus Speech Emotion RecognitionManagement consulting / professional services AI adoption strategy

Concept

A vendor-delivered engagement for professional-services and other knowledge-work firms that replaces opinion-based AI policy with measurement. The vendor works with the client to (1) sample 15–20 real, representative task types from live client work (e.g., market sizing, slide production, data analysis, memos on ambiguous evidence); (2) recruit 50–100 of the firm's own consultants; (3) randomly assign them to no-AI, AI-assisted, or AI-assisted-plus-training conditions, mirroring the published field-experiment design; and (4) blind-grade all outputs on a quality rubric while tracking task counts and completion times. The deliverable is a task-level frontier map: for each task type, the measured effect of AI on volume, speed, and quality — green (AI helps), red (AI measurably hurts correctness) — plus a one-page AI-use policy per task type. No platform or tooling is sold; the map and policy are the product.

Why now

The field experiment [0] proves both the method and the need: in a preregistered study with 758 BCG consultants across 18 realistic tasks, AI-assisted consultants completed 12.2% more tasks, 25.1% faster, with significantly higher quality — yet on a complex managerial task deliberately placed outside the AI frontier, AI users were 19% less likely to produce correct solutions. Crucially, the frontier is "jagged": tasks of seemingly similar difficulty fall on opposite sides, so intuition and vendor marketing cannot tell a firm which of its tasks are safe. Older corpus evidence [4] reinforces this: across 39 clinical benchmarks, LLMs that ace knowledge exams show poorly characterized translation to practice — public benchmark scores do not transfer to real work, so firms must measure on their own tasks.

Written by GLM · 9 Oct 2026

AI assessment

Backed by 2 papers92

A high-conviction, service-based wedge that monetizes the 'jagged frontier' finding by providing empirical risk-mapping for high-stakes professional services.

Evidence strength
5/5
The idea directly operationalizes the methodology and findings of the BCG field experiment, supported by a second paper confirming the gap between benchmark and practice performance.
Market pull
4/5
Top-tier consulting firms have high urgency to integrate AI but face extreme reputational risk if AI-generated errors reach clients, making a 'safety map' highly valuable.
Novelty & moat
4/5
While similar to the 'Rapid Local Effectiveness Trials' for health, this applies a specific experimental design (randomized trials on tasks) to a different high-value sector with a unique 'jagged frontier' problem.
Feasibility
5/5
The MVP is a service engagement requiring no proprietary software development, only the application of an existing experimental protocol.
Wedge clarity
5/5
The 6-week fixed-scope engagement is a sharp, low-friction entry point that delivers a tangible asset (the map) without requiring a long-term software contract.
Simplicity / focus
5/5
The product is a single, well-defined diagnostic service rather than an over-scoped platform or toolset.

Scored by Gemma 4 · 9 Oct 2026. Rated against a fixed rubric (evidence, market, novelty, feasibility, wedge, simplicity), relative to the closest ideas already in the catalog. A prior estimate to compare ideas before real-world signal arrives.

Persona discussion

AI personas trained on real people's expertise debate this idea as it evolves.

View the discussion →

Act on this idea

Ideas only matter if someone runs with them. Your message goes straight to the founder's inbox — nothing is stored on our servers.

Who benefits

  • Boston Consulting Groupcompany

    Its own 758 consultants were the experimental subjects; running the same trial design across its global offices would let BCG set practice-level AI policies from its own measured data instead of blanket rules.

  • McKinsey & Companycompany

    Employs tens of thousands of consultants on exactly the task types tested and has already deployed internal generative AI tools; a task-level frontier map tells it where those tools raise versus degrade delivery quality.

  • Deloittecompany

    A large consulting and audit workforce spans many service lines; measured per-task AI effects would let leadership set differentiated AI-use policies rather than one firmwide rule.

  • Bain & Companycompany

    Faces the same AI rollout decisions for its consultant base; the trial design provides evidence for where to require human-only work.

  • Accenturecompany

    Could use trials internally and offer the measured-frontier method within its AI-transformation engagements, replacing survey-based readiness assessments with hard effect sizes.

Research it builds on

  1. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality
    Fabrizio Dell’Acqua, Edward McFowland, Ethan Mollick et al. · 2026 · 170 citations
    All ideas from this paper →
  2. Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks
    Eun Jeong Gong, Chang Seok Bang, Jae Jun Lee et al. · 2025 · 30 citations
    All ideas from this paper →
Leave feedback
feasibility