Beyond subjective judgments: Predicting evaluations of creative writing from computational linguistic features

Abstract

The question of how to evaluate creativity in the context of creative writing has been a subject of ongoing discussion. A key question is whether something as elusive as creativity can be evaluated in a systematic way that goes beyond subjective judgments. To answer this question, we tested whether human evaluations of the creativity of short stories can be predicted by: (1) established measures of creativity and (2) computerized linguistic analyses of the stories. We conducted two studies, in which college students (with and without interest and experience in creative writing) wrote short stories based on a writing prompt. Independent raters (six in Study 1, five in Study 2) assessed the stories using an evaluation rubric specifically designed to assess aspects of creativity, on which they showed high interrater reliability. We provide evidence of convergent validity, in that the rubric evaluations correlated with established creativity measures, including measures of divergent thinking, associative fluency, and self-reported creative behavior and achievements. Linguistic properties of the short stories were analyzed with two computerized text analysis tools: Coh-Metrix, which analyzes aspects of text cohesion and readability, and Linguistic Inquiry and Word Count, which identifies meaningful psychological categories of the text content. Linguistic features predicted the human ratings of creativity to a significant degree. These results provide novel evidence that creative writing can be evaluated reliably and in a systematic way that captures objective features of the text. The results further establish our evaluation rubric as a useful tool to assess creative writing. Keywords Creativity. Creative writing. Language. Coh-Metrix. Linguistic inquiry and word count (LIWC) Creative writing is a form of expression that affords great freedom. Writers are constrained only by the limits of their imagination (and sometimes a deadline). Writing is also very personal. Good writers are thought to develop an authentic Bvoice^or writing style that makes their work unique and recognizable among others. Confirming this common perception, research has shown that students in a writing class were able to identify both famous authors and even their peers from samples of their writing (Gabora, O'Connor, & Ranjan, 2012). This uniqueness of expression is an aspect of creative writing that is highly valued, but it poses a challenge when it comes to evaluating the creativity of a piece of writing. If every piece of writing is unique, can we nonetheless compare different pieces * Claire M. Zedelius

Automating creativity assessment with SemDis: An open platform for computing semantic distance

“…To address the limitations of subjective scoring, researchers have begun to explore the utility of automated scoring approaches using computational tools (Acar & Runco, 2014;Dumas et al, 2020;Dumas & Runco, 2018;Green, 2016;Hass, 2017b;Heinen & Johnson, 2018;Kenett, 2019;Prabhakaran et al, 2014;Zedelius, Mills, & Schooler, 2019). One such approach uses latent semantic analysis (LSA; Landauer, Foltz, & Laham, 1998) to quantify the "semantic distance" between concepts in a given semantic space.…”

Automating Creativity Assessment with SemDis: An Open Platform for Computing Semantic Distance

“…To address the limitations of subjective scoring, researchers have begun to explore the utility of automated scoring approaches using computational tools (Acar & Runco, 2014;Dumas et al, 2020;Dumas & Runco, 2018;Green, 2016;Hass, 2017b;Heinen & Johnson, 2018;Kenett, 2019;Prabhakaran et al, 2014;Zedelius, Mills, & Schooler, 2019). One such approach uses Latent Semantic Analysis (LSA; Landauer, Foltz, & Laham, 1998) to quantify the "semantic distance" between concepts in a given semantic space.…”

Measuring Creative Writing with the Storyboard Task: The Role of Effort and Story Length

“…But product‐based assessments also capture multiple domain‐general creative abilities (e.g., generating ideas and evoking mental representations, respectively; e.g., Barbot et al, 2013) as well as domain‐specific nuances (e.g., Kaufman & Baer, 2002). Therefore, it is important to examine the role of effort in more holistic (i.e., product‐based), ecologically valid assessments of creativity, such as in creative writing (Forgeard & Kaufman, 2015; Zedelius et al, 2019).…”