Automating Creativity Assessment with SemDis: An Open Platform for Computing Semantic Distance

Abstract

Creativity research requires assessing the quality of ideas and products. In practice, conducting creativity research often involves asking several human raters to judge participants’ responses to creativity tasks, such as judging the novelty of ideas from the alternate uses task (AUT). Although such subjective scoring methods have proved useful, they have two inherent limitations—labor cost (raters typically code thousands of responses) and subjectivity (raters vary on their perceptions and preferences)—raising classic psychometric threats to reliability and validity. We sought to address the limitations of subjective scoring by capitalizing on recent developments in automated scoring of verbal creativity via semantic distance, a computational method that uses natural language processing to quantify the semantic relatedness of texts. In five studies, we compare the top performing semantic models (e.g., GloVe, continuous bag of words) previously shown to have the highest correspondence to human relatedness judgements. We assessed these semantic models in relation to human creativity ratings from a canonical verbal creativity task (AUT; Study 1-3) and novelty/creativity ratings from two word association tasks (Study 4-5). We find that a latent semantic distance factor—comprised of the common variance from five semantic models—reliably and strongly predicts human creativity and novelty ratings across a range of creativity tasks. We also replicate an established experimental effect in the creativity literature (i.e., the serial order effect) and show that semantic distance correlates with other creativity measures, demonstrating convergent validity. We provide an open platform to efficiently compute semantic distance, including tutorials and documentation (semdis.wlu.psu.edu).

Existem concordância e confiabilidade na avaliação da criatividade de resultados tangíveis da aprendizagem de computação na Educação Básica?

“…Assim, uma possibilidade para pesquisas futuras é automatizar parte da avaliação da criatividade a fim de apropriar-se do progresso recente na avaliação automatizada para mitigar limitações relacionadas a subjetividade e tempo [61]. Assim, suportando sua aplicação amplamente no ensino de computação na Educação Básica, considerando especificamente a falta de professores bem treinados nesta área de conhecimento [7].…”

Cortical Networks of Creative Ability Trace Gene Expression Profiles of Synaptic Plasticity in the Human Brain

“…To objectively quantify the creative quality of responses, we used several computational models of semantic distance (Beaty and Johnson, 2020). Semantic distance captures the novelty facet of creativity by computing the cosine similarity of concepts in large corpora of natural language (Prabhakaran et al, 2014;Kenett, 2019;Beaty and Johnson, 2020).…”

“…To objectively quantify the creative quality of responses, we used several computational models of semantic distance (Beaty and Johnson, 2020). Semantic distance captures the novelty facet of creativity by computing the cosine similarity of concepts in large corpora of natural language (Prabhakaran et al, 2014;Kenett, 2019;Beaty and Johnson, 2020). We computed semantic distance using an online application called SemDis, an open platform developed to automate creativity assessment via semantic distance 1 (Beaty and Johnson, 2020).…”

“…Semantic distance captures the novelty facet of creativity by computing the cosine similarity of concepts in large corpora of natural language (Prabhakaran et al, 2014;Kenett, 2019;Beaty and Johnson, 2020). We computed semantic distance using an online application called SemDis, an open platform developed to automate creativity assessment via semantic distance 1 (Beaty and Johnson, 2020). SemDis leverages five compositional vector models to compute the relatedness between inputted texts: three continuous bag of words (CBOW) predict models and two count models.…”

How Reliable and Valid are Frequency-Based Originality Scores? More Evidence from a Two-Step Approach

“…Scores based on the frequency of occurrence of ideas are probably most commonly used. Some of the most recent ways for the scoring of DT tests employ semantic scoring algorithms (e.g., Beaty & Johnson, 2020;Beketayev & Runco, 2016;Dumas et al, in press).…”