Are ratings in the eye of the beholder? A non-technical primer on many facet Rasch measurement to evaluate rater effects on teacher behavior rating scales

Styck, Kara M.; Anthony, Christopher J.; Flavin, A.; Riddle, David; LaBelle, Brittany

doi:10.1016/j.jsp.2021.01.001

Cited by 7 publications

(8 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite the conceptualization of DBR as more objective than standard behavior rating scales, "rater effects" (variability in assigned ratings that is due to the rater and not the student being rated 1 ; Engelhard & Wind, 2017) have been documented to be present for DBR between raters using GT studies (e.g., Christ et al, 2010) just as they are for other indirect behavior assessment methods (e.g., teacher behavior rating scales; Styck et al, 2021). Various recommendations have been made to address between-rater differences in DBR as a result of this GT research through standardizing administration procedures, such as, only using DBR for intrapersonal decisions (e.g., progress monitoring), training raters prior to using DBR (e.g., Briesch et al, 2010), and keeping the rater constant.…”

Section: Methods For Considering Rater Effectsmentioning

confidence: 99%

“…However, another analytic approach to considering rater effects has yet to be applied to DBR scores. Specifically, the many-facet Rasch measurement (MFRM; Linacre, 2018) approach offers several advantages for consideration of rater effects (e.g., Styck et al, 2021) including providing the opportunity for a finer-grained consideration of the scope and nature of rater effects as well as their individual impact on scores. Thus, the purpose of this study was to apply MFRM to DBRs to evaluate the scope, nature, and impact of rater effects on DBR scores.…”

Section: Direct Behavior Ratingsmentioning

confidence: 99%

“…From this perspective, differences in raters’ general tendencies is not viewed as random error, but rather systematic variation, and measurement models will try to “make sense” of this systematic variation (i.e., by estimating parameters to quantify it) similar to items in the previous example. Such analyses hold many advantages for evaluating rater effects including procedures for determining the nuanced ways in which individuals’ rating behavior differs (e.g., enabling consideration of how raters differentially utilize scale categories; Styck et al, 2021), methods for evaluating specific rater effects (e.g., central tendency effects; Myford & Wolfe, 2003), and procedures for correcting for differences in rater severity/leniency.…”

Section: Direct Behavior Ratingsmentioning

confidence: 99%

“…Indeed, MFRM has only recently been applied to other commonly used school-based behavioral assessments. Specifically, two studies by Anthony et al (2022) and Styck et al (2021) evaluated the presence of rater effects on brief versions of the Social Skills Improvement System-Teacher Rating Scale (Gresham & Elliott, 2008) and the Academic Competence Evaluation Scales (DiPerna & Elliott, 2000). These studies revealed that rater effects were variegated and different in form across different constructs and item formats.…”

Section: Rationale and Purposementioning

confidence: 99%

See 3 more Smart Citations

Using many-facet rasch measurement and generalizability theory to explore rater effects for direct behavior rating–multi-item scales.

Anthony

Styck²,

Volpé³

et al. 2023

School Psychology

Self Cite

View full text Add to dashboard Cite

Although originally conceived of as a marriage of direct behavioral observation and indirect behavior rating scales, recent research has indicated that Direct Behavior Ratings (DBRs) are affected by rater idiosyncrasies (rater effects) similar to other indirect forms of behavioral assessment. Most of this research has been conducted using generalizability theory (GT), yet another approach, many-facet Rasch measurement (MFRM), has recently been utilized to illuminate the previously opaque nature of these rater idiosyncrasies. The purpose of this study was to utilize both approaches (GT and MFRM) to consider rater effects with 126 second-through fifth-grade students who were rated on two DBR-Multi-Item Scales by four raters (22 of these ratings were fully crossed). Results indicated the presence of rater effects and revealed nuances about their nature, including showing differences across construct domains, identifying items that are potentially more susceptible to rater effects than others, and isolating specific raters who appear to have been more susceptible to rater effects than other raters. These findings further indicate the indirect nature of DBRs and offer potential avenues for addressing and ameliorating rater effects in research and practice. Impact and ImplicationsOur study examined whether scores on a popular behavioral assessment, Direct Behavior Ratings-Multi-Item Scales (DBR-MIS), were affected by rater effects, which refer to score differences attributable to rater characteristics rather than student behavior. We used two methodologies that enable a fine-grained examination of these rater effects. Our results illuminated nuanced features of DBR-MIS that hold promise to better address rater effects in future research and practice.

show abstract

Section: Methods For Considering Rater Effectsmentioning

confidence: 99%

Section: Direct Behavior Ratingsmentioning

confidence: 99%

Section: Direct Behavior Ratingsmentioning

confidence: 99%

Section: Rationale and Purposementioning

confidence: 99%

See 2 more Smart Citations

Using many-facet rasch measurement and generalizability theory to explore rater effects for direct behavior rating–multi-item scales.

Anthony

Styck²,

Volpé³

et al. 2023

School Psychology

Self Cite

View full text Add to dashboard Cite

show abstract

“…The rating data were analyzed using Many-Facet Rasch Measurement (MFRM) with the software FACETS (Version No.3.82.2) (Linacre, 2018b ). MFRM is an analytic approach in the rater-mediated assessment that allows for fine-grained evaluation of the rating behavior of individual raters and estimation of student proficiency while adjusting for differences in rater variability (Styck et al, 2021 ). It also allows examination of other aspects of the rating scenario, such as the rubric, and interaction between factors (Myford and Wolfe, 2003 , 2004 ).…”

Section: Methodsmentioning

confidence: 99%

Assessment as Learning: How Does Peer Assessment Function in Students' Learning?

2022

View full text Add to dashboard Cite

Peer assessment (PA) is employed as one fundamental practice of classroom-based assessment in terms of its learning-oriented and formative nature. The exercise of peer assessment has multiple and additional benefits for student learning. However, research into the learning processes in peer assessment is scarce both in theory and in practice, making it difficult to evaluate and pinpoint its value as a tool in assessment as learning (AaL). This study focuses both on the learning process and outcome through assessment activities. We set out with three goals in mind: (1) to examine students' assessment performance in context, (2) to evaluate its impact on student progress, and (3) to illuminate teachers on organizing assessment activities. Three specific research questions are answered in this study: (1) How do student raters perform in the process of PA in an advanced English class? (2) To what extent do assessment activities influence the students' speaking ability? (3) What are students' perceptions of PA regarding its benefits and caveats? A total of 29 undergraduate students participated in two assessment activities on argumentative speaking. Many-Facet Rasch Model analysis was conducted to measure the rater effects both at the group level and the individual level. Bias/interaction analyses were performed to diagnose rater behavior in different contexts including the rating session, speaking session, and peer assessment vs. self-assessment. Questionnaire and semi-structured interview data were also collected to explore factors and strategies that could interfere with PA as AaL. Results show that students exhibited stable rating behavior and made progress in argumentative speaking in all dimensions, including delivery, organization, and language use. They are more stringent with themselves than with peers although there is one rare case with a bias against peers. Participants acknowledged the benefits of PA but also shared reasonable concerns in practice. This study validated the feasibility and the effectiveness of PA for student learning. Discussion on findings and guidelines for effective implementation of PA as AaL are provided.

show abstract

What is the concordance between parent- and education professional-reported adaptive functioning in autistic children using the VABS-II?

Moore¹,

Couteur

Charman

et al. 2022

J Autism Dev Disord

View full text Add to dashboard Cite

Adaptive functioning of autistic children is traditionally measured through informant-report, often from parents. Behaviour varies across settings though, and context-specific reports should be considered. Limited and inconsistent results show low parent-education professional concordance, but no research has yet explored item level response variation. We investigated Vineland Adaptive Behaviour Scales-II concordance using 233 lower ability autistic children from the PACT-G sample. Domain and item level agreement was low, but better on objectively measured behaviours. Higher child nonverbal ability improved concordance. Where disagreements occurred, education professionals identified emergent skills more and parents were more likely to rate present/absent. Parents and education professionals view the adaptive abilities of autistic children differently and both should be considered when developing personalised interventions and support.

show abstract

Are ratings in the eye of the beholder? A non-technical primer on many facet Rasch measurement to evaluate rater effects on teacher behavior rating scales

Cited by 7 publications

References 46 publications

Using many-facet rasch measurement and generalizability theory to explore rater effects for direct behavior rating–multi-item scales.

Using many-facet rasch measurement and generalizability theory to explore rater effects for direct behavior rating–multi-item scales.

Assessment as Learning: How Does Peer Assessment Function in Students' Learning?

What is the concordance between parent- and education professional-reported adaptive functioning in autistic children using the VABS-II?

Contact Info

Product

Resources

About