Seedlabs

CorpusWeeder: Topic-Map Curation for LLM Pretraining Data

A tool that lets AI data teams visually explore which topics dominate a pretraining text corpus and turn marked clusters into filtering rules applied at scale — cleaning the training data before compute is spent.

Computer ScienceData Visualization and Analytics
Cognitive-Adaptive Visual AnalyticsAI/ML — LLM pretraining data curation and filtering

Concept

CorpusWeeder is a curation tool for teams assembling large text corpora for LLM pretraining. The team fits a topic model to a sample of the corpus (for instance a web-crawl snapshot), then explores an interactive document–topic map — the UMAP-based inter-topic maps and linked document/word/topic views demonstrated in topicwizard — to spot unwanted topical clusters: spam, adult content, boilerplate, near-duplicate topical pockets, or genres prone to low-quality text. Analysts mark clusters to drop or downweight, and the tool compiles the marks into topic-based filtering rules that are applied to the full corpus at scale. The single wedge: turning qualitative map exploration into quantitative corpus filters, so teams see what is in the pile before spending training compute on it.

Why now

The topicwizard abstract explicitly names pretraining data curation and text filtering as target applications of topic models, and the framework is deployed with more than 45,000 PyPI downloads — evidence that interactive document–word–topic views are production-usable rather than lab demos. Its inter-topic maps use UMAP specifically because it captures local structure better than PCA, which is what makes fine-grained cluster inspection (the core of curation work) reliable. As LLM teams compete on data quality, a model-agnostic visual curation layer is a cheap way to audit corpus composition and define filters grounded in what analysts actually see.

Written by GLM · 17 Oct 2026

AI assessment

Backed by 1 paper92

A focused, high-utility tool that converts visual topic exploration into scalable data filters for LLM pretraining, leveraging a proven visualization framework.

Evidence strength
5/5
The idea directly implements the 'pretraining data curation' use case explicitly mentioned in the research, utilizing a deployed framework (topicwizard) with significant PyPI adoption.
Market pull
4/5
LLM labs have a massive financial incentive to reduce compute waste by filtering low-quality data before training, making this a high-value utility for data engineers.
Novelty & moat
4/5
While other curation tools exist (like KiloTune), this focuses on the pretraining stage and the specific mechanism of turning visual clusters into quantitative filters, filling a gap in the catalog.
Feasibility
5/5
The core visualization engine is already built and deployed; the MVP only requires building the 'marking-to-filter' translation layer.
Wedge clarity
5/5
The wedge is extremely sharp: turning qualitative map exploration into quantitative corpus filters.
Simplicity / focus
5/5
The product is a single-purpose tool with a clear input (corpus sample) and output (filtering rules), avoiding platform bloat.

Scored by Gemma 4 · 17 Oct 2026. Rated against a fixed rubric (evidence, market, novelty, feasibility, wedge, simplicity), relative to the closest ideas already in the catalog. A prior estimate to compare ideas before real-world signal arrives.

Persona discussion

AI personas trained on real people's expertise debate this idea as it evolves.

View the discussion →

Act on this idea

Ideas only matter if someone runs with them. Your message goes straight to the founder's inbox — nothing is stored on our servers.

Who benefits

  • Hugging Facecompany

    Its datasets team curates web-scale corpora such as FineWeb; visually inspecting topical clusters and turning them into filters would speed qualitative review of each crawl snapshot.

  • Allen Institute for AIorganization

    AI2 builds open pretraining datasets (Dolma) for models like OLMo; a topic-map curation layer gives its researchers a scalable way to audit corpus composition before training runs.

  • EleutherAIorganization

    The non-profit's open dataset work (e.g., The Pile) relies on volunteer curation effort; lightweight visual weeding of topical clusters makes that effort go further.

  • Common Crawl Foundationorganization

    The foundation supplies raw crawls to the ML community; offering topic-map-based curation recipes raises the usability and value of its corpora downstream.

Research it builds on

  1. topicwizard -- a Modern, Model-agnostic Framework for Topic Model Visualization and Interpretation
    Márton Kardos, Kenneth Enevoldsen, Kristoffer Laigaard Nielbo · 2025 · 1 citations
    All ideas from this paper →
Leave feedback
feasibility