Application of Latent Semantic Analysis to Divergent Thinking is Biased by Elaboration

Abstract

Scoring divergent‐thinking response sets has always been challenging because such responses are not only open‐ended in terms of number of ideas, but each idea may also be expressed by a varying number of concepts and, thus, by a varying number of words (elaboration). While many current studies have attempted to score the semantic distance in divergent‐thinking responses by applying latent semantic analysis (LSA), it is known from other areas of research that LSA‐based approaches are biased according to the number of words in a response. Thus, the current article aimed to identify and demonstrate this elaboration bias in LSA‐based divergent‐thinking scores by means of a simulation. In addition, we show that this elaboration bias can be reduced by removing the stop words (for example, and, or, for and so forth) prior to analysis. Furthermore, the residual bias after stop word removal can be reduced by simulation‐based corrections. Finally, we give an empirical illustration for alternate uses and consequences tasks. Results suggest that when both stop word removal and simulation‐based bias correction are applied, convergent validity should be expected to be highest.

Automating Creativity Assessment with SemDis: An Open Platform for Computing Semantic Distance

“…In addition, prior research suggests one substantial weakness of applying an additive compositional model in creativity assessment is that it penalizes (i.e., reduces) semantic distance scores for more elaborate creativity responses (Forthmann et al, 2018). We attempt to replicate this finding and determine whether or not multiplicative models similarly penalize semantic distance scores, with the goal of explaining maximal variance in human creativity ratings.…”

“…Notably, our study used multiplicative models, whereas Dumas and colleagues used additive models, and we found substantially better prediction of human ratings compared to additive models-a finding that replicated in Study 2 and is consistent with recent work comparing additive and multiplicative models in the context of predicting human similarity judgments . In addition, Forthmann et al, (2018) showed that additive compositional models penalize, that is, reduce the semantic distance of longer creative responses, when in fact elaboration should often increase creativity. While removing stop words mitigates this penalty (Forthmann et al, 2018), for the first time, we showed that a multiplicative compositional model reversed the correlation between elaboration and semantic distance: a multiplicative model showed positive correlations between semantic distance and elaboration and an additive model showed negative correlations.…”

“…In addition, Forthmann et al, (2018) showed that additive compositional models penalize, that is, reduce the semantic distance of longer creative responses, when in fact elaboration should often increase creativity. While removing stop words mitigates this penalty (Forthmann et al, 2018), for the first time, we showed that a multiplicative compositional model reversed the correlation between elaboration and semantic distance: a multiplicative model showed positive correlations between semantic distance and elaboration and an additive model showed negative correlations. More work will be needed to more systematically investigate the role elaboration bias may still play in semantic distance 2 , but we show that employing a multiplicative model substantially improved the correspondence between human creativity ratings and semantic distance.…”

“…The initial LSA findings of Prabhakaran et al, (2014) have since been replicated using a different computational model and corpora (Heinen & Johnson, 2018) and extended to other creativity tasks, including the AUT (Hass, 2017b), albeit with mixed evidence for validity (Forster & Dunbar, 2009;Forthmann, Oyebade, Ojo, Günther, & Holling, 2018;Harbison & Haarmann, 2014;Hass, 2017bHass, , 2017a. As LSA has been increasingly employed in creativity research, researchers have begun to identify limitations of the approach and best-practices in data processing.…”

“…Prior research that used additive composition models found that responses with higher word counts received a penalty in semantic distance, meaning lower semantic distance scores (Forthmann et al, 2018).…”