A Model and its Fit lie in the Eye of the Beholder: Long Live the Sum Score

Edelsbrunner, Peter Adriaan

doi:10.31234/osf.io/8tq7p

Cited by 1 publication

(1 citation statement)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In considering a possible future applied scenario in which the MOTES item scores would be summed or averaged to produce a total score either for the entire MOTES or for a particular scale (Edelsbrunner, 2022; McNeish & Wolf, 2020), we calculated the internal consistency reliability via Cronbach’s alpha (Cronbach, 1951). The entire set of 24 items was highly reliable (α = .92, 95% confidence intervals [CI] [.91, .93]), and the three scales also reached satisfactory levels of internal consistency (uses = 0.81, 95% CI [.78, .84]; examples = 0.83, 95% CI [.80, .85]; sentences = 0.86, 95% CI [.84, .88]).…”

Section: Phase 5—internal and External Validation Of Motesmentioning

confidence: 99%

Measuring original thinking in elementary school: Development and validation of a computational psychometric approach.

Acar,

Dumas,

Organisciak

et al. 2024

Journal of Educational Psychology

View full text Add to dashboard Cite

Creativity is highly valued in both education and the workforce, but assessing and developing creativity can be difficult without psychometrically robust and affordable tools. The open-ended nature of creativity assessments has made them difficult to score, expensive, often imprecise, and therefore impractical for school-or district-wide use. To address this challenge, we developed and validated the Measure of Original Thinking for Elementary School (MOTES) in five phases, including the development of the item pool and test instructions, expert validation, cognitive pilots, and validation of the automated scoring and latent test structure. MOTES consists of three game-like computerized activities (uses, examples, and sentences subscales), with eight items in each for a total of 24 items. Using large language modeling techniques, MOTES is scored for originality by our open-access artificial intelligence platform with a high level of agreement with independent subjective human ratings across all three subscales at the response level (rs = .79, .91, and .85 for uses, examples, and sentences, respectively). Confirmatory factor analyses showed a good fit with three factors corresponding to each game, subsumed under a higher-order originality factor. Internal consistency reliability was strong for both the subscales (H = 0.82, 0.85, and 0.88 for uses, examples, and sentences, respectively) and the higher-order originality factor (H = 0.89). MOTES scores showed moderate positive correlations with external creative performance indicators as well as academic achievement. The implications of these findings are discussed in relation to the challenges of assessing creativity in schools and research. Educational Impact and Implications StatementThis study introduces Measure of Original Thinking for Elementary School (MOTES), a validated assessment tool consisting of three game-like computerized activities, to address the challenge of assessing and nurturing creativity in education. By employing large language modeling methods, MOTES demonstrates remarkable correlations with human ratings, representing a significant leap forward in computational psychometrics for creativity assessment. The automated scoring methods employed by MOTES overcome cost-ineffectiveness and lack of affordability issues associated with traditional creativity tests, offering a reliable and affordable means of measuring original thinking in the verbal domain. These scores can be utilized alongside other indicators of school learning and academic achievement, highlighting the potential for creativity assessment to play a pivotal role in schoolbased programming and development efforts.

show abstract

Section: Phase 5—internal and External Validation Of Motesmentioning

confidence: 99%