Seedlabs
Source research

Collective Critics for Creative Story Generation

Minwook Bae, Hyounghun Kim · 2024 · 9 citationsRead the paper

Generating a long story of several thousand words with narrative coherence using Large Language Models (LLMs) has been a challenging task.Previous research has addressed this challenge by proposing different frameworks that create a story plan and generate a long story based on that plan.However, these frameworks have been mainly focusing on maintaining narrative coherence in stories, often overlooking creativity in story planning and the expressiveness of the stories generated from those plans, which are desirable properties to captivate readers' interest.In this paper, we propose Collective Critics for Creative Story Generation framework (CRITICS), which is composed of plan refining stage (CRPLAN) and story generation stage (CRTEXT), to integrate a collective revision mechanism that promotes those properties into long-form story generation process.Specifically, in each stage, a group of LLM critics and one leader collaborate to incrementally refine drafts of plan and story throughout multiple rounds.Extensive human evaluation shows that the CRITICS can significantly enhance story creativity and reader engagement, while also maintaining narrative coherence.Furthermore, the design of the framework allows active participation from human writers in any role within the critique process, enabling interactive human-machine collaboration in story writing. 1

What the full text shows

Working prototype
  • CRPLAN significantly improves story plan interestingness, coherence, and creativity compared to the DOC pipeline. — Interesting: 85.00% vs 57.56%, Coherent: 77.89% vs 67.33%, Creative: 84.33% vs 57.33% (Human evaluation of 300 pairwise comparisons using gpt-3.5-turbo)
  • CRTEXT significantly enhances story interestingness and creativity compared to the DOC pipeline. — 'significantly higher performance' (exact percentages not provided in text) (Human evaluation of 300 stories using gpt-3.5-turbo)
  • The framework maintains effectiveness across different LLM backbones and baselines. — 'superior performance' in Creativity and Coherence vs Re3 baseline (Streamlined human evaluation using GPT-4 backbone)
  • Increasing revision rounds beyond 3 leads to higher inventiveness but reduces narrative coherence. — 'significant changes' vs 'significantly detracts from coherence' (CRPLAN stage)
Setup
300 premises from the DOC pipeline; human evaluation by three inter-annotators; comparisons against DOC and Re3 pipelines using gpt-3.5-turbo and GPT-4.
How
The CRITICS framework uses a two-stage process (CRPLAN for planning and CRTEXT for generation) where a group of LLM critics and one leader collaboratively refine drafts through multiple rounds of persona-driven critique. It employs specific creative criteria (Image, Voice, Originality) to guide the revision process.
Limitations
Effectiveness is tied to the capabilities of the specific LLMs used. · Currently optimized for English language applications. · Criteria are limited to those defined by Mozaffari (2013). · Risk of introducing toxicity or falsehoods in the text.

Summarized by Gemma 4 from the open-access full text · 10 Oct 2026. Read it