Detecting Differential Rater Functioning in Severity and Centrality: The Dual DRF Facets Model

Abstract

Performance assessments heavily rely on human ratings. These ratings are typically subject to various forms of error and bias, threatening the assessment outcomes’ validity and fairness. Differential rater functioning (DRF) is a special kind of threat to fairness manifesting itself in unwanted interactions between raters and performance- or construct-irrelevant factors (e.g., examinee gender, rater experience, or time of rating). Most DRF studies have focused on whether raters show differential severity toward known groups of examinees. This study expands the DRF framework and investigates the more complex case of dual DRF effects, where DRF is simultaneously present in rater severity and centrality. Adopting a facets modeling approach, we propose the dual DRF model (DDRFM) for detecting and measuring these effects. In two simulation studies, we found that dual DRF effects (a) negatively affected measurement quality and (b) can reliably be detected and compensated under the DDRFM. Using sample data from a large-scale writing assessment ( N = 1,323), we demonstrate the practical measurement consequences of the dual DRF effects. Findings have implications for researchers and practitioners assessing the psychometric quality of ratings.

A sequential approach to detecting differential rater functioning in sparse rater-mediated assessment networks

“…In future studies, researchers could consider an ANOVA-based sequential approach to detecting DRF under various conditions. In addition, researchers may also consider the use of the recently proposed DDRF model (Jin & Eckes, 2021) for detecting and adjusting for differential rater severity and centrality effects alongside the sequential approach and to compare the efficiency of these two methods to detect DRF under a variety of conditions. Researchers could consider the alignment between the sequential approach and scale purification procedures (Magis & Facon, 2013; W.-C. Wang et al, 2009) to further explore the conditions under which the sequential approach can accurately identify artificial and real DRF.…”

“…Only recently have researchers proposed methods to statistically control for DRF. Specifically, Jin and Eckes (2021) proposed the Dual Differential Rater Functioning (DDRF) model that allows researchers to detect and control for differential rater severity and rater centrality. This approach is promising, and additional research is needed to explore its use in language testing research.…”

Potential scoring and predictive bias in interim and summative writing assessments.

“…Third, generalization inferences require that students’ writing performance not be specific to a particular testing time, score type, or rater. The latter is a consistent concern with scoring student responses (e.g., Jin & Eckes, 2022).…”

“…When directly evaluating students’ written compositions, human raters have shown systematic biases such as assigning different scores to students of diverse backgrounds, changing their application of rubric criteria over time, and overusing the middle score values (Huang, 2012; Jin & Eckes, 2022). Getting consistent rater agreement can be particularly challenging with holistic rubrics due to the unclear weighting that raters place on different traits, or qualities or aspects of the writing, when assigning the overall score (Ohta et al, 2018).…”

“…Relatedly, generalization inferences in Kane's (2013) framework require that students' writing performance not be specific to a particular testing time, score type, or rater. The latter is a consistent concern with scoring student responses (e.g., Jin & Eckes, 2022). Our study used different raters and score types, but that may not be feasible for schools to do, especially if administering multiple prompts over time.…”