Scoring Creativity Tests By Computer

Fifty years later and still working: Rediscovering Paulus et al’s (1970) automated scoring of divergent thinking tests.

“…The validity was also stronger as compared with semantic distance. Importantly, using the Paulus et al (1970) features in a state-of-the-art supervised machine learning approach does not outperform the simple stepwise regression used by Paulus et al (1970). Yet, we found that supervised machine learning can outperform the Paulus et al (1970) approach, when semantic distance is added to the set of prediction variables.…”

“…Importantly, using the Paulus et al (1970) features in a state-of-the-art supervised machine learning approach does not outperform the simple stepwise regression used by Paulus et al (1970). Yet, we found that supervised machine learning can outperform the Paulus et al (1970) approach, when semantic distance is added to the set of prediction variables. We discuss challenges that are expected for future research that aim at combining unsupervised approaches based on word embeddings and supervised learning relying on text-mining features.…”

“…Using such approaches for automated scoring of divergent thinking responses has been proposed and tested approximately fifteen years ago (Forster & Dunbar, 2009). Hence, it came to our surprise when we rediscovered the work of Paulus et al (1970) that implemented an automated scoring of the Torrance Test of Creative Thinking (TTCT; Torrance, 1966) more than 50 years ago (Greene, 1971; Paulus et al, 1970; Paulus & Renzuli, 1968). Their technical report has never been cited according to Google Scholar, Greene’s 1971 conference contribution was cited once, and even the article by Paulus and Renzuli (1968) published in Gifted Child Quarterly in which all basic ideas for their automated scoring algorithm were outlined has been cited only three times according to Google Scholar (all citation counts were accurate as of September 7, 2021).…”

“…Most of the debated approaches are unsupervised machine learning approaches and researchers seemingly just started to evaluate supervised approaches. Hence, rediscovering the seminal work of Paulus et al (1970) came as a big surprise to us. More than 50 years ago, they derived prediction formulas for an automated scoring of the Torrance Test of Creative Thinking (Torrance, 1966) that was based on a set of text mining variables (e.g., average word length, word counts, and so forth).…”

“…This work reintroduces Paulus et al’s (1970) approach and investigates how it performs compared with semantic distance scoring. The main contribution of Paulus et al’s (1970) neglected masterpiece on divergent thinking assessment is echoed by the findings of this work: Creative quality of responses can be well predicted by means of simple text mining statistics. The validity was also stronger as compared with semantic distance.…”