STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation

Abstract

Systems for story generation are asked to produce plausible and enjoyable stories given an input context. This task is underspecified, as a vast number of diverse stories can originate from a single input. The large output space makes it difficult to build and evaluate story generation models, as (1) existing datasets lack rich enough contexts to meaningfully guide models, and (2) existing evaluations (both crowdsourced and automatic) are unreliable for assessing long-form creative text. To address these issues, we introduce a dataset and evaluation platform built from STORIUM, an online collaborative storytelling community. Our author-generated dataset contains 6K lengthy stories (125M tokens) with fine-grained natural language annotations (e.g., character goals and attributes) interspersed throughout each narrative, forming a robust source for guiding models. We evaluate language models fine-tuned on our dataset by integrating them onto STORIUM, where real authors can query a model for suggested story continuations and then edit them. Automatic metrics computed over these edits correlate well with both user ratings of generated stories and qualitative feedback from semistructured user interviews. We release both the STORIUM dataset and evaluation platform to spur more principled research into story generation.

Creative Natural Language Generation

“…While the tutorial will include our own work (Yao et al, 2019;He et al, 2019;Mittal et al, 2022;Goldfarb-Tarrant et al, 2020;Chakrabarty et al, 2020bAkoury et al, 2020;Stowe et al, 2021;Tian et al, 2021;Padmakumar and He, 2022;Chakrabarty et al, 2022a;Yang et al, 2022), we anticipate that roughly 40% of the tutorial content will be pulled from work by other researchers in NLP and machine learning communities include but not limited to (Ghazvininejad et al, 2016;Fan et al, 2018bFan et al, , 2019Van de Cruys, 2020;Riedl and Young, 2010;Lin and Riedl, 2021;Brahman and Chaturvedi, 2020;Mirowski et al, 2023;.…”

TaleStream: Supporting Story Ideation with Trope Knowledge

“…Other methods generate stories by adapting higher-level attributes that can consist of goals and events as story units [62,72]. Akoury et al derive a large corpus of story components from STORIUM, an online collaborative game that lets users write stories based on cards as the framework [6]. With TaleStream, we leverage tropes from the wiki tvtrope.org as building blocks of stories.…”

“…The task of generating compelling story ideas is difficult to formulate and evaluate because it is inherently subjective. Our work focuses on ensuring coherence, a key characteristic that has been identified and extensively used in previous literature [6,7,19,67]. The suggested story ideas should fit seamlessly within the user's narrative, i.e.…”

“…Efforts in developing writing support systems have focused on continuing stories by producing text blocks to add and edit [56,65]. However, beyond sentence generation, previous research has suggested that the main strength and use cases of such tools lie in their suggestive power to help overcome writer's block [6,17,24,75]. In this regard, the focus of our work is to build a system that supports story writing by providing inspiring materials.…”

Controlled Cue Generation for Play Scripts

“…Brahman et al [2020] focused on the task of interactive story generation, where the user provides mid-level sentence abstractions in the form of cue phrases to the model during the generation process. Akoury et al [2020] proposed another story generation system called STORIUM, where human authors query a model for suggested story continuations and edit them.…”