Testing Computational Assessment of Idea Novelty in Crowdsourcing

Wang, Kai; Dong, Boxiang; Ma, Junjie

doi:10.1080/10400419.2023.2187544

Cited by 3 publications

(2 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Although considered moderate by Koo and Li’s (2016) interrater reliability standards, it impacts some of the study’s external validity conclusions. As such, further research is needed to improve the way of rating creativity of AQT responses, especially via laypersons (Hass et al, 2018; Wang et al, 2023). Perhaps more effort is needed in training raters to increase the reliability and improve the AQT’s validity as a tool used to examine creativity, similar to Amabile’s (1982) consensual assessment technique or utilization of quasi-expert raters (Kaufman & Baer, 2012; Kaufman et al, 2013).…”

Section: Discussionmentioning

confidence: 99%

The role of asking more complex questions in creative thinking.

Raz,

Reiter-Palmon,

Kenett

2023

Psychology of Aesthetics, Creativity, and the Arts

View full text Add to dashboard Cite

Section: Discussionmentioning

confidence: 99%

The role of asking more complex questions in creative thinking.

Raz,

Reiter-Palmon,

Kenett

2023

Psychology of Aesthetics, Creativity, and the Arts

View full text Add to dashboard Cite

“…Валуева и др. Автоматическая оценка вербальной креативности а об идеях, выраженных в виде целых текстов (Wang et al, 2023). Конкурентами контекстно-независимых моделей дистрибутивной семантики в исследованиях творчества очень быстро стали модели, основанные на нейросетях глубокого обучения, позволяющие не только учитывать частоту встречаемости слов, но и распознавать значение слов в зависимости от контекста.…”

Section: большие языковые моделиunclassified

Automatic Scoring of Verbal Divergent Thinking Tests: From Lexical Databases to Large Language Models

Valueva,

Panfilova,

Rafikova

2024

psychology

View full text Add to dashboard Cite

The article explores the evolution of methods for automatically assessing verbal divergent thinking tests. Resear - chers have increasingly focused on the ability to evaluate the originality of respondents' answers by calculating their semantic distance from the stimulus task. From 2009 to 2019, latent semantic analysis became the primary method for assessing semantic distances. Overall, in terms of internal consistency and correlation with expert ratings, its application yielded satisfactory results, maintaining an acceptable balance of quality and effort expended. However, issues emerged (dependence on the corpus used, result instability, systematic distortions related to the length of analyzed responses), prompting researchers to transition to more advanced models of distributional semantics (GloVe, Word2Vec etc.), large language models, and supervised learning. Large language models, especially those fine-tuned on creativity test materials, demonstrated higher effectiveness compared to models assessing semantic distances and approached expert evaluations. In addition to evaluating originality, the article considers works proposing methods for automatic assessment of elaboration, flexibility, associative flow, and divergent semantic integration. References to online platforms that allow for automatic assessments of originality in responses to divergent tests are provided. The issue of interpreting results obtained through large language models is discussed. A drawback of using these models is the lack of understanding of the basis on which judgments of the originality of creative products are made. The perspectives of applying explainable artificial intelligence for evaluating results of verbal and non-verbal tests of creative thinking are being discussed.

show abstract

The Impact of Crowdsourcing on Innovation and Decision-Making

Akbari

2024

The Road to Outsourcing 4.0

View full text Add to dashboard Cite

Testing Computational Assessment of Idea Novelty in Crowdsourcing

Cited by 3 publications

References 72 publications

The role of asking more complex questions in creative thinking.

The role of asking more complex questions in creative thinking.

Automatic Scoring of Verbal Divergent Thinking Tests: From Lexical Databases to Large Language Models

The Impact of Crowdsourcing on Innovation and Decision-Making

Contact Info

Product

Resources

About