Improving Creativity Performance Assessment: A Rater Effect Examination with Many Facet Rasch Model

Hung, Su Pin; Chen, Po Hsi; Chen, Hsueh Chih

doi:10.1080/10400419.2012.730331

Cited by 28 publications

(25 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These findings are consistent with the suggestion that the observed variability in performance assessment arises from the complexity of performance in real-life setting; some authors therefore use complexity theory to explain why and how performance in critical safety areas is too complex to be reliably and accurately assessed (Bergström, Henriqson, & Dahlström, 2011). On the other hand, studies of rater performance that varied the task constraints and therefore task complexity did not lead to differences in the quality and accuracy of ratings (Hung et al, 2012). A study of workplace-based assessment in the medical field does conclude that there are differences in the cognitive processes between experienced and non-experienced raters of performance in complex environments (Goevarts, Schuwrith, van der Vleuten, & Muijtjens, 2011).…”

Section: Introductionsupporting

confidence: 86%

Peer Assessment of Aviation Performance: Inconsistent for Good Reasons

Roth

Mavin

2014

Cognitive Science

View full text Add to dashboard Cite

Research into expertise is relatively common in cognitive science concerning expertise existing across many domains. However, much less research has examined how experts within the same domain assess the performance of their peer experts. We report the results of a modified think-aloud study conducted with 18 pilots (6 first officers, 6 captains, and 6 flight examiners). Pairs of same-ranked pilots were asked to rate the performance of a captain flying in a critical pre-recorded simulator scenario. Findings reveal (a) considerable variance within performance categories, (b) differences in the process used as evidence in support of a performance rating, (c) different numbers and types of facts (cues) identified, and (d) differences in how specific performance events affect choice of performance category and gravity of performance assessment. Such variance is consistent with low inter-rater reliability. Because raters exhibited good, albeit imprecise, reasons and facts, a fuzzy mathematical model of performance rating was developed. The model provides good agreement with observed variations.

show abstract

Section: Introductionsupporting

confidence: 86%

Peer Assessment of Aviation Performance: Inconsistent for Good Reasons

Roth

Mavin

2014

Cognitive Science

View full text Add to dashboard Cite

show abstract

“…These methods provide more powerful tools to study the aspects of creativity that could not be understood otherwise. For example, Rasche modeling analyzed rater effects in assessing creativity performance that were mostly examined by interrater reliability in previous studies (Hung, Chen, & Chen, 2012). In addition, TSC and PACA, as two new journals, not only published more qualitative and mixed-methods studies but also showed more methodological rigor in these studies.…”

Section: Discussionmentioning

confidence: 99%

An Empirical Review of Research Methodologies and Methods in Creativity Studies (2003–2012)

Long

2014

Creativity Research Journal

103

View full text Add to dashboard Cite

Based on the data collected from 5 prestigious creativity journals, research methodologies and methods of 612 empirical studies on creativity, published between 2003 and 2012, were reviewed and compared to those in gifted education. Major findings included: (a) Creativity research was predominantly quantitative and psychometrics and experiment were the most frequently utilized quantitative methodologies, (b) judges were employed frequently to assess creativity and correlational techniques were utilized most widely to analyze quantitative data, (c) case study was the most frequently used qualitative methodology, (d) most mixed-methods studies were rooted in quantitative methodology, and (e) both creativity and gifted education research were dominated by quantitative methodologies, but there were less qualitative studies and slightly more mixed-methods studies on creativity. Implications of these findings were further discussed and future research directions were suggested.

show abstract

“…Recently, the MFRM's applications have appeared on not only traditional education, but also other research fields. For example, on studies of creative writing (Bardot et al, 2012), creativity (Hung et al, 2012), scale job analysis (Wang & Stahl, 2012), food behavioral analysis (Vianello & Robusto, 2010), and medical performance assessment (McManus et al, 2013).…”

Section: Many-facet Rasch Model (Mfrm)mentioning

confidence: 99%

A Comparison of Generalizability Theory and Many Facet Rasch Measurement in an Analysis of Mathematics Creative Problem Solving Test

Lee¹,

Cha²

2016

J. Curric. Eval.

View full text Add to dashboard Cite

This study describes the use of Generalizability Theory (GT) and Many-Facet Rasch Measurement (MFRM) to evaluate and improve the rating procedure in a mathematics creative problem solving test. Results indicate that these two methods agree about the relative degrees of variation among the facets but slightly differ on how to account for the sources of variation. For both the GT and MFRM results, the variance component for the Person by Item interaction is relatively large, indicating significant variability. Results from both methods also indicated that variance due to rater and interactions related with rater were relatively low. The reliability of the mean rating for each examinee based on five items, four raters and four rating criteria using a fully crossed design was 0.58(G-coefficient) and 0.49(phi coefficient). We found the guidelines from the Decision study (D-study) to obtain a more optimal reliability coefficients, it needed at least ten items. Depending on the purpose of a particular study, GT or MFRM may be the appropriate measurement technique to use.

show abstract

Improving Creativity Performance Assessment: A Rater Effect Examination with Many Facet Rasch Model

Cited by 28 publications

References 33 publications

Peer Assessment of Aviation Performance: Inconsistent for Good Reasons

Peer Assessment of Aviation Performance: Inconsistent for Good Reasons

An Empirical Review of Research Methodologies and Methods in Creativity Studies (2003–2012)

A Comparison of Generalizability Theory and Many Facet Rasch Measurement in an Analysis of Mathematics Creative Problem Solving Test

Contact Info

Product

Resources

About