How Reliable and Valid are Frequency-Based Originality Scores? More Evidence from a Two-Step Approach

Abstract

Creativity—as any other object of scientific endeavor—requires a sound measurement that adheres to quality criteria. For decades, creativity science has been criticized as falling short in developing valid and reliable measures of creative potential, activity, and achievement. Recent years have witnessed growth of theoretical and empirical works that focused on improving creativity assessment. Here, we apply one of such recently developed approaches based on item response theory to examine ideas’ and person score reliability in a divergent thinking task. A large sample (N = 621) of children and adolescents solved the Circles task from Torrance Tests of Creative Thinking-Figural and two other figural tests measuring creative thinking (Test of Creative Thinking-Drawing Production) and creative imagination (Test of Creative Imagery Abilities). By employing response propensity models, we observed that separate ideas’ reliability tended to fall below recommended thresholds (even liberal ones, e.g., .60) unless the sample size as well as the number of generated ideas (fluency) were large. Importantly, reliability at the idea-level affected reliability at the person-level much less than could be assumed based on recent findings. We propose a systematic perspective on divergent thinking assessment that considers responses as nested in tasks and tasks as nested in tests. Finally, we recommend that adding more tasks to divergent thinking tests might increase reliability at the task-level.