Seedlabs

Rank-Test Reporting Audit for Journal Submissions

A tool for journal editors to check if the statistics in a submitted paper are reported correctly.

A manuscript-screening tool for editorial offices that finds every Mann-Whitney, Wilcoxon signed-rank, Kruskal-Wallis, and Friedman result in a submission and flags missing effect sizes, mismatched measures, and magnitude labels that contradict the published cutoffs — a statcheck for rank tests.

Decision SciencesMeta-analysis and systematic reviews
Materials Science Information SystemsScholarly publishing — editorial office and peer-review triage

Concept

A text- and table-mining checker that runs on manuscripts before peer review. It detects the four common nonparametric tests in results text and tables, extracts the reported statistics (U, W, H, chi-square, n, p), recomputes the standardized effect size where the inputs allow, and returns an editorial triage report with three flag types: (1) a rank test reported with no effect size at all; (2) the wrong measure attached to the test, e.g. Cohen's d beside a Mann-Whitney U; (3) a verbal magnitude inconsistent with the cutoffs — an r of 0.25 called 'large' when large starts at 0.5, or a VDA of 0.60 dismissed as negligible when small begins at 0.56. Editors get a two-minute statistical triage before assigning review, and authors get a pre-submission self-check.

Why now

The review states that effect-size estimation for nonparametric tests 'remains less explored' than for parametric tests — which is exactly why rank-test results in submissions so often appear as bare p-values — and it consolidates the audit rulebook: the correct measure per test and numeric cutoffs for all six measures (r at 0.1/0.3/0.5; VDA at 0.56/0.64/0.71; rank-biserial at 0.11/0.28/0.43; ordinal eta-squared at 0.01/0.06/0.14; ordinal epsilon-squared at 0.01/0.08/0.26; Kendall's W at 0.1/0.3/0.5). Automated checking of rank-test reporting was impractical before because no single consolidated standard existed to check against.

Written by GLM · 10 Oct 2026

AI assessment

Backed by 1 paper89

A highly focused, technically feasible audit tool that solves a specific, documented pain point in scholarly publishing by automating the verification of nonparametric effect sizes.

Evidence strength
5/5
The idea directly operationalizes the specific thresholds and test-measure pairings provided in the full-text digest of the research paper.
Market pull
4/5
Journal editors and professional editing services have a strong incentive to reduce desk-rejection cycles and improve statistical rigor.
Novelty & moat
4/5
While similar to 'TheoryDesk Screen' in workflow, it targets a distinct, quantitative technical gap (nonparametric reporting) rather than qualitative theory.
Feasibility
4/5
The scope is limited to a few specific tests and a fixed set of numeric rules, making it a straightforward NLP and regex-based extraction task.
Wedge clarity
5/5
The product has a very sharp entry point: a 'statcheck' specifically for rank tests, avoiding the trap of a general-purpose statistical auditor.
Simplicity / focus
5/5
It is a single-purpose tool with one clear function: flagging reporting errors in rank-test results.

Scored by Gemma 4 · 10 Oct 2026. Rated against a fixed rubric (evidence, market, novelty, feasibility, wedge, simplicity), relative to the closest ideas already in the catalog. A prior estimate to compare ideas before real-world signal arrives.

Persona discussion

AI personas trained on real people's expertise debate this idea as it evolves.

View the discussion →

Act on this idea

Ideas only matter if someone runs with them. Your message goes straight to the founder's inbox — nothing is stored on our servers.

Who benefits

  • SciScorecompany

    SciScore already automates rigor and reporting audits of submitted manuscripts for publishers; a rule-based nonparametric effect-size check is a concrete checklist addition no competing audit tool has.

  • American Journal Experts (AJE)company

    AJE sells statistical review and editing to authors; the audit gives its reviewers a fast, consistent triage of nonparametric reporting before human effort is spent.

  • PLOS ONEorganization

    PLOS ONE's submission volume includes many rank-based analyses on small or skewed samples; an automated effect-size screen enforces a stated reporting expectation at scale without adding reviewer burden.

  • Cactus Communications (Editage)company

    Editage's statistical services could offer a pre-submission nonparametric effect-size check as a distinct service grounded in the published cutoffs.

Research it builds on

  1. Effect sizes for nonparametric tests
    Fernanda Fiel Peres · 2025 · 149 citations
    All ideas from this paper →
Leave feedback
feasibility