Dissecting Reliability and Validity Evidence of Subjective Creativity Assessment: A Literature Review

Long, Haiying; Wang, Jue

doi:10.1007/s10648-022-09679-0

Cited by 13 publications

(12 citation statements)

References 97 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Product-based assessments require individuals to construct a response. The most commonly used product-based tests of creative thinking in education are arethe Torrance Test of Creative Thinking (TTCT; Torrance 1966 , 1990 ) and variants such as the Abbreviated Torrance Test for Adults (ATTA; Goff 2002 ), as well as assessments by Guilford ( 1967 ) and Wallach and Kogan ( 1965 ) ( Long and Wang 2022 ). The OECD also developed a Creative Thinking assessment for PISA 2022 which was administered in over 60 countries.…”

Section: Conceptualizations Of Creative Thinkingmentioning

confidence: 99%

“…The product-based response format allows participants to construct their responses, and generally include domain agnostic stimuli to limit the influence of prior knowledge on performance. These components allow for a broad range of both expression of ideas for participants and opportunities for inspiration; however, Long and Wang ( 2022 ) found that most of these tasks rely on alternative use tasks in which participants brainstorm to identify alternative uses for common objects and that those objects were limited to mainly a brick, box, or knife (e.g., Lee and Therriault 2013 ).…”

Section: Conceptualizations Of Creative Thinkingmentioning

confidence: 99%

“…The scoring of product-based responses also has significant limitations. Human-scoring of student responses requires training, a level of expertise, and inherently involves a degree of subjectivity ( Long and Wang 2022 ; Silvia et al 2008 ). Inconsistencies also exist among the level of expertise required, the number of raters per item, and the thresholds for inter-rater reliability.…”

Section: Conceptualizations Of Creative Thinkingmentioning

confidence: 99%

“…Inconsistencies also exist among the level of expertise required, the number of raters per item, and the thresholds for inter-rater reliability. Perhaps most problematic for the scoring of creative thinking assessments are the inconsistencies and range of criteria for scoring skills common across assessments as well as the range of inclusion of skills and the weight given to those skills for scoring ( Long and Wang 2022 ). The challenge of scoring creative thinking at an international scale was achieved by the 2022 PISA Creative Thinking assessment through multiple extensive coder trainings, training materials, and query services provided to address coder queries ( OECD 2023 in press).…”

Section: Conceptualizations Of Creative Thinkingmentioning

confidence: 99%

“…The limitations of current assessments, noted above, create challenges for scalability and inclusion in the classroom ( Long and Wang 2022 ). To answer these challenges and to facilitate the development of this essential skillset, we propose specific assessment design recommendations based on the use of the assessment.…”

Section: Creative Thinking and Innovation: Measurementmentioning

confidence: 99%

See 4 more Smart Citations

Navigating the Spectrum of Conventionality: Toward a New Model of Creative Thinking

Stoeffler

Daley

2023

J. Intell.

View full text Add to dashboard Cite

Current conceptualizations of creative thinking focus primarily on the measurement of creative thinking for the purpose of identifying creative thinking proficiency. We propose a conceptualization that includes a framework and assessments that focus on the measurement and learning of creative thinking and innovation skills. Our conceptualization involves an understanding that innovation is a critical application of creative thinking and that the process of creative thinking that leads to innovation can be performed intentionally and explicitly. In this paper, we put forth a process model for creative thinking and innovation that focuses on an expanded set of cognitive and social skills and processes that facilitate the navigation of the spectrum of conventionality. The process model includes the conventional thinking skill, which serves as not only a foundational skill for understanding and navigating the spectrum of conventionality, but also facilitates the reliable measurement of creative thinking and innovation by supporting the generation of a response pool that represents the full spectrum of conventionality for use in scoring. We explore the advantages of this model and how it addresses some of the challenges presented by current creative thinking conceptualizations and assessments. Finally, we explore the implications of implementing this process model for education.

show abstract

Section: Conceptualizations Of Creative Thinkingmentioning

confidence: 99%

Section: Conceptualizations Of Creative Thinkingmentioning

confidence: 99%

Section: Conceptualizations Of Creative Thinkingmentioning

confidence: 99%

Section: Conceptualizations Of Creative Thinkingmentioning

confidence: 99%

Section: Creative Thinking and Innovation: Measurementmentioning

confidence: 99%

See 3 more Smart Citations

Navigating the Spectrum of Conventionality: Toward a New Model of Creative Thinking

Stoeffler

Daley

2023

J. Intell.

View full text Add to dashboard Cite

show abstract

What Makes Children's Responses to Creativity Assessments Difficult to Judge Reliably?

Dumas,

Acar,

Berthiaume

et al. 2023

Journal of Creative Behavior

View full text Add to dashboard Cite

Open‐ended verbal creativity assessments are commonly administered in psychological research and in educational practice to elementary‐aged children. Children's responses are then typically rated by teams of judges who are trained to identify original ideas, hopefully with a degree of inter‐rater agreement. Even in cases where the judges are reliable, some residual disagreement on the originality of the responses is inevitable. Here, we modeled the predictors of inter‐rater disagreement in a large (i.e., 387 elementary school students and 10,449 individual item responses) dataset of children's creativity assessment responses. Our five trained judges rated the responses with a high degree of consistency reliability (α = 0.844), but we undertook this study to predict the residual disagreement. We used an adaptive LASSO model to predict 72% of the variance in our judges' residual disagreement and found that there were certain types of responses on which our judges tended to disagree more. The main effects in our model showed that responses that were less original, more elaborate, prompted by a Uses task, from younger children, or from male students, were all more difficult for the judges to rate reliably. Among the interaction effects, we found that our judges were also more likely to disagree on highly original responses from Gifted/Talented students, responses from Latinx students who were identified as English Language Learners, or responses from Asian students who took a lot of time on the task. Given that human judgments such as these are currently being used to train artificial intelligence systems to rate responses to creativity assessments, we believe understanding their nuances is important.

show abstract

Are Some People Generally More Creative Than Others? A Systematic Review of Fifty Years’ Research

Miravete,

Tricot

2024

Educ Psychol Rev

View full text Add to dashboard Cite

Dissecting Reliability and Validity Evidence of Subjective Creativity Assessment: A Literature Review

Cited by 13 publications

References 97 publications

Navigating the Spectrum of Conventionality: Toward a New Model of Creative Thinking

Navigating the Spectrum of Conventionality: Toward a New Model of Creative Thinking

What Makes Children's Responses to Creativity Assessments Difficult to Judge Reliably?

Are Some People Generally More Creative Than Others? A Systematic Review of Fifty Years’ Research

Contact Info

Product

Resources

About