Seedlabs

Raincheck: Per-User Raincloud Readouts for A/B Tests

A tool for A/B testers that shows every user's individual result to find hidden patterns that averages miss.

A feature for experimentation platforms that shows each A/B test's result as per-variant raincloud plots of every user's value, so teams can spot segment-split or bimodal effects that a single 'average lift' number hides.

Computer ScienceData Visualization and Analytics
Cognitive-Adaptive Visual AnalyticsSaaS product analytics / A/B testing — experiment result readouts.

Concept

Every experimentation dashboard answers "did B beat A?" with a mean lift and a confidence interval — but a +2% average can be +20% for power users and negative for everyone else, and the topline looks identical. Raincheck is a readout renderer for experimentation platforms: for each experiment metric, it draws per-variant raincloud plots of the per-user values — raw dots (sample size and outliers visible), a box summary, and a violin for distribution shape. Variants are factorial groups, and a continuous covariate (tenure, session count, spend) can be mapped into the display to expose segment structure. The renderer flags experiments where variant distributions differ in shape — bimodality, heavy tails, distinct outlier clusters — even when the mean-based test is flat, turning "no significant result" into "look at the segments." Platforms already log per-user metric values, so this is a readout-layer change, not a data-pipeline change.

Why now

The plotting primitive is deployed on CRAN (ggrain's single geom_rain function) with exactly the required capabilities — factorial grouping for variant comparison and mapping a secondary continuous covariate for segment exposure — so the renderer is an adaptation, not an invention. The failure mode it targets is documented in the corpus: [1] (Gatherplot) shows that aggregation-based displays such as histograms and violins alone discard per-case detail, and summary-only experiment readouts do the same at decision time. No mainstream experimentation tool currently renders per-user distributions as a standard readout, leaving the wedge open.

Written by GLM · 11 Oct 2026

AI assessment

Backed by 2 papers88

A high-utility visualization plugin for A/B testing platforms that solves the 'flaw of averages' by surfacing bimodal or segmented effects through raincloud plots.

Evidence strength
5/5
The idea directly applies the demonstrated capabilities of the ggrain package (factorial grouping and covariate mapping) to a specific business problem, supported by the Gatherplot research on the pitfalls of aggregation.
Market pull
4/5
Experimentation platforms (Statsig, Eppo) are in a feature war to provide deeper 'why' insights beyond p-values, and their target users (Data Scientists/PMs) actively struggle with hidden segment effects.
Novelty & moat
3/5
While raincloud plots exist in academia and TrialRain applies them to pharma, applying them as a standard readout for SaaS A/B testing is a distinct and valuable application.
Feasibility
5/5
Since the plotting logic is already implemented in R/ggplot2 and the data (per-user metrics) is already collected by these platforms, this is a frontend rendering task rather than a data engineering challenge.
Wedge clarity
5/5
The wedge is extremely sharp: replace or augment the standard 'mean lift' chart with a distribution-aware readout to find hidden winners in 'flat' tests.
Simplicity / focus
5/5
The product is a single, focused visualization tool with one clear purpose, avoiding the trap of becoming a general-purpose analytics platform.

Scored by Gemma 4 · 11 Oct 2026. Rated against a fixed rubric (evidence, market, novelty, feasibility, wedge, simplicity), relative to the closest ideas already in the catalog. A prior estimate to compare ideas before real-world signal arrives.

Persona discussion

AI personas trained on real people's expertise debate this idea as it evolves.

View the discussion →

Act on this idea

Ideas only matter if someone runs with them. Your message goes straight to the founder's inbox — nothing is stored on our servers.

Who benefits

  • Statsigcompany

    Statsig's platform already logs per-user metric values for every experiment; adding raincloud readouts is a readout-layer feature that differentiates its results pages and surfaces segment-split effects its topline lift numbers currently hide.

  • Eppocompany

    Eppo positions itself on statistical rigor for data teams; distribution-honest readouts extend that positioning and help customers avoid shipping 'average wins' that mask segment losses.

  • Amplitudecompany

    Amplitude's Experiment add-on serves product teams deciding rollouts; per-user raincloud views would let them see bimodal or outlier-driven effects behind flat topline results.

  • Optimizelycompany

    Optimizely's results pages are summary-first; per-user distribution views address a long-standing blind spot where heterogeneous effects look like null results.

  • Booking.comcompany

    Booking.com runs one of the largest industrial experimentation programs; its product teams would catch guest-segment-split effects earlier, in the readout they already look at.

Research it builds on

  1. Gatherplot: A Non-Overlapping Scatterplot
    Deokgun Park, Sung-Hee Kim, Niklas Elmqvist · 2024 · 1 citations
    All ideas from this paper →
  2. ggrain – a R package for raincloud plots
    Nicholas Judd, Jordy van Langen, Davide Poggiali et al. · 2024 · 1 citations
    All ideas from this paper →
Leave feedback
feasibility