Ill-Structured Problem-Solving Assessment Builder
An authoring and scoring tool that lets districts and test publishers build ill-structured math problem-solving tasks — missing information, multiple defensible answers — and score both the solution and the solving process against validated rubrics.
Concept
An authoring and scoring tool for assessments of genuine mathematical problem solving, aimed at districts, test publishers, and research programs. It provides a library of ill-structured task templates — problems with missing information that must be requested, problems with multiple defensible answers, problems solvable by several strategies — plus built-in process capture: students log their strategy choice and complete short self-monitoring checkpoints while working, and the tool scores both the solution and the process evidence against validated rubrics. The output is a comparable problem-solving score that survives the move from routine exercises to ill-structured tasks, with an audit trail of what was actually measured. The single wedge: making ill-structured, process-scored items as easy to deploy and defend as multiple-choice.
Why now
Three chapters of the volume converge on assessment as the bottleneck: the chapter on treating mathematics as an ill-structured discipline [0], the chapter on assessing problem solving with its short-term remedy and long-term solution [0], and the chapter on testing mathematical problem solving [0] together establish that routine items systematically fail to measure the construct. The volume's metacognition chapter [0] supplies the missing piece — understanding what you are doing is itself something students can be asked to demonstrate — so self-monitoring can be scored, not just observed. Older corpus work on developing and validating teaching assessment criteria [5] shows that rubrics for complex, judgment-heavy practices can be built and reliability-tested, providing the methodological precedent for validating process-scoring rubrics here.
Written by GLM · 3 Oct 2026
AI assessment
A specialized authoring tool for high-stakes assessment publishers to transition from routine math testing to validated, process-oriented problem-solving metrics.
- Evidence strength 4/5
- The idea is strongly grounded in a convergence of research regarding ill-structured disciplines and the necessity of assessing metacognition, though the link to MBI:TAC is a methodological analogy rather than direct evidence.
- Market pull 4/5
- Major assessment bodies (PISA, ETS) have a high urgency to modernize 'problem solving' metrics to reflect real-world competency, providing a clear B2B buyer profile.
- Novelty & moat 3/5
- While it shares themes with the 'Problem-Diet Audit' and 'RubricLens', it moves from auditing/general PBL to a specific, scalable authoring and scoring engine for math.
- Feasibility 4/5
- The core product is a structured CMS with rubric-based scoring logic, which is technically straightforward to build as a prototype.
- Wedge clarity 5/5
- The wedge is exceptionally sharp: replacing the 'multiple-choice' ease of deployment with a 'process-scored' equivalent for ill-structured items.
- Simplicity / focus 5/5
- The product is focused on a single, clear function: authoring and scoring a specific type of assessment task.
Scored by Gemma 4 · 11 Oct 2026. Rated against a fixed rubric (evidence, market, novelty, feasibility, wedge, simplicity), relative to the closest ideas already in the catalog. A prior estimate to compare ideas before real-world signal arrives.
Persona discussion
AI personas trained on real people's expertise debate this idea as it evolves.
View the discussion →Act on this idea
Ideas only matter if someone runs with them. Your message goes straight to the founder's inbox — nothing is stored on our servers.
Business analysis
The PESTEL analysis reveals a strong alignment with current educational shifts toward metacognition and complex problem-solving, though it faces significant friction from the legacy of standardized, multiple-choice testing. While technologically feasible, the primary hurdles are the legal and political risks associated with the 'defensibility' of non-binary scoring in high-stakes environments.
Political2
Economic2
Social2
Technological2
Environmental2
Legal2
The success of a new assessment standard depends heavily on educational policy, government regulations, and the social shift toward measuring complex competencies over rote memorization. · Generated 2026-10-11 by Gemma 4AI-generatedFull PESTEL Analysis →
Who benefits
- ETSorganization
Its core business is building and validating large-scale assessments; a validated authoring-and-scoring approach for ill-structured problem solving addresses a long-standing construct-validity gap in its math products.
- OECD (PISA)organization
PISA's mathematics framework centers real-world problem solving; a builder for ill-structured, process-scored tasks operationalizes that framework for national and international item development.
- Pearsoncompany
Its assessment division develops item banks for many states; templates and rubrics for process-scored problem-solving items would let it produce the next-generation items states are beginning to demand.
- College Boardorganization
Its math assessments face recurring criticism that items are routine; a defensible path to ill-structured, process-evidenced items would strengthen its construct claims.
Research it builds on
- The Teaching and Assessing of Mathematical Problem SolvingRandall Charles, Edward A. Silver · 2025 · 333 citationsAll ideas from this paper →
- Development and Validation of the Mindfulness-Based Interventions – Teaching Assessment Criteria (MBI:TAC)Rebecca Crane, Catrin Eames, Willem Kuyken et al. · 2013 · 265 citationsAll ideas from this paper →
- Treating Mathematics as an Ill-Structured DisciplineLauren Β. Resnick · 2025 · 120 citationsAll ideas from this paper →
- Real-time cognitive and emotional state tracking in intelligent tutoring systems for enhanced learning outcomesKai Kong, Haytham F. Isleem, Rajanikanth Aluvalu et al. · 2025 · 8 citationsAll ideas from this paper →
- Parental Involvement, Homework, and Self-RegulationHéfer Bembenutty · 2006 · 5 citationsAll ideas from this paper →
- Encouraging Metacognitive Reflection through Prompts in a Computer-Based Learning Platform: Failure to Find a Benefit in a Large-Scale Randomized TrialAllison Zengilowski, Brendan A. Schuetze, Abubakir Siedahmed et al. · 2025 · 3 citationsAll ideas from this paper →