Using many-facet rasch measurement and generalizability theory to explore rater effects for direct behavior rating–multi-item scales.

Abstract

Although originally conceived of as a marriage of direct behavioral observation and indirect behavior rating scales, recent research has indicated that Direct Behavior Ratings (DBRs) are affected by rater idiosyncrasies (rater effects) similar to other indirect forms of behavioral assessment. Most of this research has been conducted using generalizability theory (GT), yet another approach, many-facet Rasch measurement (MFRM), has recently been utilized to illuminate the previously opaque nature of these rater idiosyncrasies. The purpose of this study was to utilize both approaches (GT and MFRM) to consider rater effects with 126 second-through fifth-grade students who were rated on two DBR-Multi-Item Scales by four raters (22 of these ratings were fully crossed). Results indicated the presence of rater effects and revealed nuances about their nature, including showing differences across construct domains, identifying items that are potentially more susceptible to rater effects than others, and isolating specific raters who appear to have been more susceptible to rater effects than other raters. These findings further indicate the indirect nature of DBRs and offer potential avenues for addressing and ameliorating rater effects in research and practice. Impact and ImplicationsOur study examined whether scores on a popular behavioral assessment, Direct Behavior Ratings-Multi-Item Scales (DBR-MIS), were affected by rater effects, which refer to score differences attributable to rater characteristics rather than student behavior. We used two methodologies that enable a fine-grained examination of these rater effects. Our results illuminated nuanced features of DBR-MIS that hold promise to better address rater effects in future research and practice.

Measurement Efficiency of a Teacher Rating Scale to Screen for Students at Risk for Social, Emotional, and Behavioral Problems

“…Even if this is a common problem in individual teacher ratings ( Anthony et al 2016 ; Wu 2017 ) and LD compromises the results of a GPCM only to a small degree ( Song 2019 ), the results might be caused by specific rater effects, such as general tendencies or halo effects ( Wu 2017 ). A potential solution might be psychometric evaluation approaches that allow to consider rater effects in behavior rating scales such as the many-facet Rasch model (see Anthony et al 2022 ) or generalizability theory (e.g., Briesch et al 2014 ). However, those approaches attempt quite strict a priori design specifications, which were not applied in the current study.…”

Multivariate Structural Equation Modeling Techniques for Estimating Reliability, Measurement Error, and Subscale Viability When Using Both Composite and Subscale Scores in Practice

“…Historically, G-theory has been applied more often to subjectively than to objectively scored measures, with recent examples spanning disciplines that include educational psychology and measurement [18][19][20][21][22], medical education [23][24][25][26][27][28][29][30][31], school psychology [32][33][34][35][36][37], classroom assessment [38,39], second language education and linguistics [40][41][42], thinking skills and creativity [43], music performance [44], athletic training [45], job performance [46], and many other areas. This makes sense because raters typically change over situations, and focal indices of reliability in G-theory (e.g., generalizability or G coefficients) represent the extent to which results can be generalized to broader domains of all possible raters.…”

Using Rasch analysis to examine raters’ expertise Turkish teacher candidates’ competency levels in writing different types of test items

“…The multi-surface Rasch approach was preferred in this context in the related research. When the rater effect is mentioned, it was examined whether the raters were experienced (Barkaoui, 2010;Davis, 2016;Erman Aslanoğlu & Şata, 2021;Kim, 2020) or the scoring rigidity within themselves (Anthony, Styck, Volpe, & Robert, 2022;Jones & Bergin, 2019;Kamış & Doğan, 2017;Primi, Silvia, Jauk, & Benedek, 2019). In this research, the effect of the field expertise of the raters was examined, which is quite significant in terms of both the examination and the result.…”