Divergent Semantic Integration (DSI): Extracting Creativity from Narratives with Distributional Semantic Modeling

Abstract

Narrative text permeates our lives from job applications to journalistic stories to works of fiction. Developing automated metrics that capture creativity in narrative text has potentially far reaching implications. Human ratings of creativity in narrative text are labor-intensive, subjective, and difficult to replicate. Across 27 different story prompts and over 3,500 short stories, we used distributional semantic modeling to automate the assessment of creativity in narrative texts. We tested a new metric to capture one key component of creativity in writing – a writer’s ability to connect divergent ideas. We termed this metric, word-to-word semantic diversity (w2w SemDiv). We compared six models of w2w SemDiv that varied in their computational architecture. The best performing model employed Bidirectional Encoder Representations Transformer (BERT), which generates context-dependent numerical representations of words (i.e., embeddings). The BERT w2w SemDiv scores demonstrated impressive predictive power, explaining up to 72% of the variance in human creativity ratings, even exceeding human inter-rater reliability for some tasks. In addition, w2w SemDiv scores generalized across Ethnicity and English language proficiency, including individuals identifying as Hispanic and L2 English speakers. We provide a tutorial with R code (osf.io/ath2s) on how to compute w2w SemDiv. This code is incorporated into an online web app (semdis.wlu.psu.edu) where researchers and educators can upload a data file with stories and freely retrieve w2w SemDiv scores.

EmoAtlas: An emotional profiling tool merging psychological lexicons, artificial intelligence and network science

“…A larger distance between words corresponding with higher creativity levels represents the same working hypothesis of the divergent semantic integration (DSI) score presented in 69 , but from a semantic, rather than syntactic, perspective.…”

“…• Text-based psychometric task, training an AI model to learn emotional and network features of texts and produce creativity ratings for each text being as close as possible to human ratings. This task is performed in English for 1071 short texts, annotated by 4 raters 69 . This task showcases how text features extracted from EmoAtlas can match BERT scores in evaluating human creativity norms rating short stories.…”

“…Here we test how EmoAtlas fares against BERT in reproducing the mean creativity ratings of short stories, collected and rated in 69 . There is a total of 1071 stories, each one rated independently by 4 human raters on a creativity scale from 1 (low creativity) to 5 (high creativity).…”

“…The latter is the dependent variable we want to train our AI on. The independent variables we attribute to each story are: BERT DSI scores (from Johnson and colleagues 69 ), emotional z-scores (from EmoAtlas) and 3 network-based features (always extracted from stories as parsed by EmoAtlas).…”

“…BERT's DSI scores were obtained by summing all cosine distances between word embeddings produced by BERT while parsing each story 69 . BERT's DSI scores correlate positively with creativity ratings (Pearson's ρ = 0.54, N = 1071, p < 10 −5 , see Fig.…”