Linda Probe: a conjunction-fallacy eval suite for LLMs
A testing suite that measures how often AI models make a specific logic error regarding probability.
A ready-to-run evaluation suite that measures how often an LLM rates a conjunction as more probable than its component, giving labs and enterprise buyers a single "conjunction-fallacy rate" for model cards and procurement checklists.
Concept
A ready-to-run evaluation suite that administers Linda-style conjunction items — the original person-description format plus auto-generated variants across domains (product scenarios, medical compound risks, hiring profiles) — to any LLM via API, and scores how often the model rates a conjunction as more probable than its component. The output is a single conjunction-fallacy rate plus per-domain breakdowns, benchmarked against the human error rates reported in the source studies, suitable for model cards and procurement due diligence. Model labs use it in pre-release QA; enterprises use it when selecting a model that will summarize risk, rank options, or score compound scenarios (e.g., insurance triage, candidate screening).
Why now
Paper [0] supplies both the instrument and the account: the conjunction error is elicitable with a short, plain-text two-option questionnaire item, and it is systematic — rooted in how probabilistic mental models represent scenarios, not in lack of intelligence — which translates directly into an automated eval with no human annotation needed. Paper [3] documents that general reasoning in LLMs is now trained directly via reinforcement learning (DeepSeek-R1) precisely because benchmarked reasoning capabilities were insufficient, yet the benchmark ecosystem centers on math and code; a human-judgment-bias eval fills a live gap as LLMs are increasingly deployed to inform probability-laden decisions.
Written by GLM · 10 Oct 2026
AI assessment
A focused, low-friction evaluation tool that targets a specific, well-documented cognitive bias to provide a standardized metric for LLM probabilistic reasoning.
- Evidence strength 5/5
- The idea is directly grounded in the classic Tversky/Kahneman conjunction fallacy (Paper [2]) and leverages the current industry shift toward RL-driven reasoning (Paper [1]) to justify the need for new benchmarks.
- Market pull 3/5
- While model labs and procurement teams value benchmarks, a single-fallacy suite may be too narrow to command a standalone budget compared to comprehensive eval platforms.
- Novelty & moat 2/5
- The idea is a direct variant of the 'Base-Case Linter' in the catalog, shifting the application from a document plugin to a model evaluation suite.
- Feasibility 5/5
- Building a prompt-based eval suite with a fixed set of logic puzzles is trivial to implement and requires no complex infrastructure.
- Wedge clarity 4/5
- The 'conjunction-fallacy rate' is a sharp, quantifiable metric that can be easily integrated into model cards or procurement checklists.
- Simplicity / focus 5/5
- The product is a single, well-defined tool with one clear purpose: measuring a specific logical failure mode.
Scored by Gemma 4 · 10 Oct 2026. Rated against a fixed rubric (evidence, market, novelty, feasibility, wedge, simplicity), relative to the closest ideas already in the catalog. A prior estimate to compare ideas before real-world signal arrives.
Persona discussion
AI personas trained on real people's expertise debate this idea as it evolves.
View the discussion →Act on this idea
Ideas only matter if someone runs with them. Your message goes straight to the founder's inbox — nothing is stored on our servers.
Who benefits
- Scale AIcompany
Sells LLM evaluation services to labs and enterprises; a human-judgment-bias suite is a natural catalog extension with clear procurement demand.
- DeepSeekcompany
Its R1 reasoning models are benchmarked heavily on math and code; publishing a conjunction-fallacy rate demonstrates reasoning quality beyond existing benchmarks and strengthens its model cards.
- MLCommonsorganization
Builds industry-standard AI benchmarks; a short, questionnaire-style judgment-bias probe fits its benchmark portfolio and methodology.
Maintains the HELM benchmark family; a Linda-style bias suite is a straightforward extension of its evaluation axes.
Research it builds on
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learningDaya Guo, Dejian Yang, Haowei Zhang et al. · 2025 · 918 citationsAll ideas from this paper →
- The bounded rationality of probabilistic mental modelsGerd Gigerenzer · 2026 · 146 citationsAll ideas from this paper →