Utilizing the Consensual Assessment Technique to Compare Creativity in Drama Spaces

Abstract

Objective: This study examines the validity of Amabile's (1982) consensual assessment technique in measuring creativity in a warm-up activity in fourth-grade drama classrooms and compares the scores between warm-ups occurring in a blackbox theater setting (experimental) vs. a traditional classroom (control). Method: Four professional actors viewed 60 clips of children's drama warm-ups and scored for creativity, using a 5-point scale. After establishing sufficient inter-rater reliability (IRR), we used the average scores of the raters to compare creativity between the experimental and control groups. Results: The raters demonstrated high agreement, with a coefficient alpha estimate of .819. An independent samples t-test between the experimental and control groups was significant at p < .001, with the experimental group receiving higher scores. Conclusions: The results suggested that creativity was significantly higher in the experimental group, and the context correlated with creativity, despite neither group having yet received drama instruction at that time. This paper presents discussions about validity, opinions of the raters, possible implications for the activity itself, and possible effect of setting on creativity.

Consensual assessment in the new domain of e-textiles: Comparing insights from expert, quasi-expert, and novice judges.

“…Additionally, the expert and quasi-expert groups both said that age affected the way they rated their projects. For example, if an artist was younger, judges tended to rate creativity higher and judge more leniently because they would not expect a young person to do such complex work, a finding that aligns with findings on using the CAT in the context of drama performance (Phonethibsavads et al, 2019). Judges across groups also aligned with general conceptions of what made artifacts less creative.…”

“…These questions are aligned with other qualitative work that aims to dig deeper into judges' subjective viewpoints and how they come to score creative products differently (e.g., Phonethibsavads et al, 2019).…”

“…Additionally, because the CAT is predicated on consensus among judges, the technique only works if judges are assessing the same construct (e.g., implicit criteria for creativity), because assessments based on incomplete representations would impede decision-making (Plucker & Makel, 2010). Therefore, valid inferences from the CAT are predicated on the idea that interrater agreement represents the judgments and values of the domain gatekeepers, which spotlights the importance of defining the expertise levels of raters (Hickey, 2001; Kaufman & Baer, 2012; Phonethibsavads et al, 2019). Although domain experts are usually thought to be the most reliable judges for creativity, some emergent domains may be lacking in history and precedent, so persons with expert-level qualifications may be scarce (Csikszentmihalyi, 1988; Glăveanu, 2010).…”

Towards a collaborative approach to measuring Social-Emotional Learning in the arts

“…Similarly, setting and achieving personal learning goals, practicing new skills, and then applying them is mastery learning, a well-researched aspect of socialemotional health (Dweck & Elliot, 1983) present in arts programs that center them (Lashley, 2018). The arts also provide opportunities to practice creativity (Phonethibsavads et al, 2019); and despite the current assessment climate in schools, there is general agreement that creativity is an important academic outcome that describes successful students in the 21st century (Sawyer, 2019).…”

Creativity Ratings of Fashion Outfits Presented on Instagram: Does Gender Matter?

“…The CAT has been applied as a methodology to measure creativity in a wide range of settings and proved to be highly reliable and valid. Reported inter-rater reliabilities mostly ranged between .70 and .90 or higher (see, e.g., Amabile, 1982bAmabile, , 1985Amabile, , 1996Baer, 1997Baer, , 1998Hennessey, 1994;Phonethibsavads, Bender, & Peppler, 2019). Numerous studies have further validated the CAT and examined some of its most prominent features, including implicit biases induced by creator characteristics (such as the ethnicity or gender of product creators; see, e.g., Kaufman, Baer, Agars, & Loomis, 2010;Kaufman, Niu, Sexton, & Cole, 2010).…”