Comparison of novice and expert evaluations of apparel design illustrations using the consensual assessment technique

Freeman, Charles; Son, Jihyeong; McRoberts, Lisa Barona

doi:10.1080/17543266.2015.1018960

Cited by 21 publications

(11 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although the standard CAT procedure requires judges to have “expertise” (e.g., Kaufman, Plucker, & Baer, 2008)—a concept still not conclusively clarified (Cseh & Jeffries, 2019), but originally associated with close familiarity with the domain in which a product was created (Amabile, 1996)—this study used nonexpert raters. Considering previous empirical evidence, this approach is consistent with the CAT: Freeman, Son, and McRoberts (2015) showed that experts and nonexperts did not differ in their creativity assessments of fashion designs. On the contrary, their ratings were actually highly correlated ( r = .83, p < .001) and matched those of experts.…”

Section: Methodssupporting

confidence: 74%

Creativity Assessment over Time: Examining the Reliability of CAT Ratings

Barth

Stadtmann

2020

Journal of Creative Behavior

View full text Add to dashboard Cite

The consensual assessment technique (CAT) is a reliable and valid method to measure (product) creativity and often considered the gold standard of creativity assessment. The reliability measure traditionally applied in CAT studies—inter‐rater reliability—cannot capture time‐sampling error, which is a particular relevant source of error for specific applications of the CAT. Therefore, the present study intended to investigate the test–retest reliability of CAT ratings. We asked raters (N = 61) for their creativity assessment of the same set of 90 fashion outfits at an initial rating session and a follow‐up session either 2 or 4 weeks later. We found that mean product ratings—the actual focus of interest in the CAT—were highly stable over time, as evidenced by consistency and agreement ICCs clearly exceeding levels of .90. However, individual raters (partially) lacked temporal stability, indicating a drift in rater tendencies over time. Our findings support the CAT’s reputation as a highly reliable measurement method, but question the temporal rating stability of the CAT’s actual “measurement instrument,” namely individual judges.

show abstract

Section: Methodssupporting

confidence: 74%

Creativity Assessment over Time: Examining the Reliability of CAT Ratings

Barth

Stadtmann

2020

Journal of Creative Behavior

View full text Add to dashboard Cite

show abstract

“…In contrast, other researchers have not found experts to always produce significantly more reliable agreement than novices. Freeman, Son, and McRoberts (2015) compared the ratings of three expert and three novice judges in rating fashion illustrations using the CAT and found no difference between the expertise groups in terms of agreement. Amabile (1982) noted that in some of her studies expertise did not seem to increase inter-rater reliability as much as expected, noting "no clear superiority of artists over nonartists in interjudge reliability" (p. 1006), while nonartist and artist judges, when rating collages, showed reasonably good inter-rater agreement.…”

Section: Suitability Of Judgesmentioning

confidence: 99%

A scattered CAT: A critical evaluation of the consensual assessment technique for creativity research.

Cseh

Jeffries

2019

Psychology of Aesthetics, Creativity, and the Arts

102

View full text Add to dashboard Cite

Amabile's Consensual Assessment Technique (CAT)taking the consensus opinions of domain expertsis considered a 'gold standard' of creativity assessment for research purposes. While several studies have identified how specific procedural choices impact on the CAT's reliability as a measure, researchers' depth of knowledge about procedures and their effects still remains incomplete. This paper explores gaps in the research by reviewing CAT and creativity literature, and aims to explore to what extent the creativity research community needs to revisit and reflect on the CAT and solidify protocols for its implementation. The conclusion highlights the need for new debate and a program of research to clarify, evidence, and harmonize CAT methodology, while simultaneously preserving the CAT's flexibility. This would enable the development and sophistication of the CAT, including possible new assistive technologies, to further strengthen its use within the science of creativity.

show abstract

“…Freeman, Son, and McRobertsʼs (2015) research on creativity assessment within fashion illustration evaluations found that "the major factor limiting further creativity development in fashion programs continues to be a lack of effective evaluation and/or evaluation tools for faculty" (p. 122). Many scholars agree that there is a need to improve evaluation techniques generally in higher education (HE; Atkinson & Lim, 2013;Franklin & Smith, 2015;Freeman et al, 2015) and yet curiously, most studies on the use of technological tools to aid assessment have focused primarily on primary and secondary school use rather than its applications within HE (Chen, Lao, & Sheum 2003, Rogers et al 2005, Sharples, Corlett, & Westmancott 2002, as cited in Franklin & Smith, 2015Nikou & Economides, 2018). Notably, Mobile-Based Assessment (MBA) is an emergent research field within the broader mobile learning area, despite a gap in literature (Nikou & Economides, 2018).…”

Section: State Of the Fieldmentioning

confidence: 99%

Forwarding fashion: developing an app to contemporize fashion design assessment and feedback in higher education

Dupuis¹

2021

Preprint

View full text Add to dashboard Cite

There is a need to evolve the current collective praxis for providing assessment and feedback on creative projects within fashion and related domains in higher education (HE) programs. Rapid advancements and disruption taking place within fashion and related creative industries signal a need for graduates who can think more divergently, while the increased diversity among current and future cohorts in HE who have different learning requirements, along with faculty facing increased time constraints and limitations in their work processes, call for new methods for providing assessment and feedback efficiently and effectively. This study proposes a practical prototype of a mobile technological solution that could help to contemporize current praxis, and may contribute to the emerging field of research: Mobile-Based Assessment (MBA).

show abstract

Comparison of novice and expert evaluations of apparel design illustrations using the consensual assessment technique

Cited by 21 publications

References 25 publications

Creativity Assessment over Time: Examining the Reliability of CAT Ratings

Creativity Assessment over Time: Examining the Reliability of CAT Ratings

A scattered CAT: A critical evaluation of the consensual assessment technique for creativity research.

Forwarding fashion: developing an app to contemporize fashion design assessment and feedback in higher education

Contact Info

Product

Resources

About