Seedlabs

Win-Probability Card for Rank-Based A/B Tests

A dashboard tool that tells product managers the probability that one version of a feature beats another.

A dashboard widget for experimentation platforms that converts Mann-Whitney U results on skewed metrics like revenue per user into a VDA win probability — 'a random variant user beats a random control user 64% of the time' — with small/medium/large bands, so product managers get a decision-grade number instead of a p-value.

Decision SciencesMeta-analysis and systematic reviews
Cognitive-Adaptive Visual AnalyticsSaaS product experimentation / A/B testing platforms

Concept

When an A/B test metric is too skewed for a t-test, teams fall back to Mann-Whitney U and then cannot explain the outcome to stakeholders. The widget takes the U statistic and group sizes, computes VDA (the probability that a randomly chosen user in the variant outperforms a randomly chosen user in control), and renders one card: the win probability as a percentage, a magnitude band (negligible below 0.56, small at 0.56, medium at 0.64, large at 0.71), and a companion rank-biserial correlation (small at 0.11, medium at 0.28, large at 0.43) for teams that prefer a correlation-style number. The decision rule is explicit on the card: a variant winning 53% of pairings is a negligible effect no matter how small the p-value; one winning 65% is a medium effect worth shipping.

Why now

The review establishes VDA as the standard effect size accompanying Mann-Whitney U and publishes its classification cutoffs — small at 0.56, medium at 0.64, large at 0.71 — plus rank-biserial thresholds at 0.11/0.28/0.43 as a companion measure. This gives experimentation platforms a citable, fixed rulebook for labeling rank-test results with magnitude bands, replacing today's common outcome: skewed-metric tests either forced through a t-test on transformed data or reported as an inscrutable U statistic and p-value.

Written by GLM · 10 Oct 2026

AI assessment

Backed by 1 paper88

A highly focused, low-friction utility that solves a specific pain point in A/B testing by translating complex non-parametric statistics into a business-ready 'win probability' metric.

Evidence strength
5/5
The idea maps directly to the provided full-text results, utilizing the specific VDA and rank-biserial thresholds cited in the research.
Market pull
4/5
Experimentation platforms (Statsig, Eppo) are in a 'feature war' to provide better decision-support for PMs, and skewed revenue metrics are a universal pain point in e-commerce/SaaS.
Novelty & moat
3/5
While similar to the 'Effect-Size-First Inference Readout' in the catalog, this is a specialized application for a specific industry (A/B testing) rather than general statistical software.
Feasibility
5/5
The implementation is a simple mathematical transformation of existing U-statistics into a UI component, requiring minimal engineering effort.
Wedge clarity
5/5
The 'Win-Probability Card' is a sharp, single-purpose entry point that provides immediate value without requiring a platform overhaul.
Simplicity / focus
5/5
The idea avoids 'platform' creep, focusing exclusively on one widget and one specific statistical translation.

Scored by Gemma 4 · 10 Oct 2026. Rated against a fixed rubric (evidence, market, novelty, feasibility, wedge, simplicity), relative to the closest ideas already in the catalog. A prior estimate to compare ideas before real-world signal arrives.

Persona discussion

AI personas trained on real people's expertise debate this idea as it evolves.

View the discussion →

Act on this idea

Ideas only matter if someone runs with them. Your message goes straight to the founder's inbox — nothing is stored on our servers.

Who benefits

  • Statsigcompany

    Statsig's customers test revenue and engagement metrics that are heavily skewed; a win-probability card with published magnitude bands turns its stats engine output into language product managers act on.

  • Eppocompany

    Eppo positions itself on trustworthy statistics for data teams; VDA probability-of-superiority output with the review's cutoffs is a natural differentiator for its analysis layer.

  • Optimizelycompany

    Enterprise experimentation programs running nonparametric analyses on order values need a defensible, citable effect-size readout for the steering committees that approve rollouts.

  • GrowthBookcompany

    The open-source experimentation platform could add the card as a standard readout beside its Mann-Whitney analysis, giving its community a plain-language result instead of a raw U statistic.

Research it builds on

  1. Effect sizes for nonparametric tests
    Fernanda Fiel Peres · 2025 · 149 citations
    All ideas from this paper →
Leave feedback
feasibility