The Rater Bundle Model

Abstract

In this article an item response model is introduced for repeated ratings of student work, which we have called the Rater Bundle Model (RBM). Development of this model was motivated by the observation that when repeated ratings occur, the assumption of conditional independence is violated, and hence current state-ofthe-art item response models, such as the rater facets model, that ignore this violation, underestimate measurement error, and overestimate reliability. In the rater bundle model these dependencies are explicitly parameterized. The model is applied to both real and simulated data to illustrate the approach.The main point of this article is to introduce a model for repeated ratings of student work, which we have dubbed the rater bundle model (RBM), and to report on its usefulness in a number of contexts. The motivation for developing this model is that we believe that the current "state-of-the-art" item response models being applied to this situation are not fully expressing our understanding of what is going on when a rater re-rates a piece of student work that has already been rated by another rater. We will call this the "repeated rating" problem. Although this is probably an observation that has been made in the past, the first time it was observed in the literature was by Patz (1996), at least as far as we are aware. Consequent to this observation, several researchers have begun developing alternate models and indices:

The Effects of Rater Severity and Rater Distribution on Examinees' Ability Estimation for Constructed‐response Items

“…Rater severity effects have been included in some measurement models and studied by DeCarlo, Kim, and Johnson (2011) ;Donoghue, McClellan, and Gladkova (2006); Engelhard (2002); Longford (1995) ;Patz, Junker, Johnson, and Mariano (2002); Wilson and Hoskens (2001); and Wolfe and Myford (1997). The results from their studies indicate that the bias or systematic error may be caused by varying degrees of rater leniency or strictness.…”

“…There are a few research studies incorporating rater severity effects into the item response theory (IRT) models (Donoghue et al, 2006;Engelhard, 1996;Patz, 1997;Patz & Junker, 1999;Wilson & Hoskens, 2001). The FACETS model (Linacre, 1991) is the IRT model that allows for the estimation of differences in severity between raters, and thus eliminates rater bias from the estimates of the items and examinees' ability.…”

“…Although there are possible mitigations of the problem under certain scenarios involving many examinees, prompts, and raters by use of techniques proposed by Haberman (1977), the solution to the problem of joint estimation generally involves the use of marginal maximum likelihood or conditional maximum likelihood. Other IRT models are also available for modeling rater effects, such as Muraki's rater effect models (1993), the hierarchical rater model (HRM model; Patz, 1997), and the rater bundle model (Wilson & Hoskens, 2001). …”

Statistical analysis of the National Institutes of Health peer review system

“…Viewed more generally, methods developed in this article extend existing statistical methodology for the analysis of multirater ordinal data (4-7) and item response data (8)(9)(10)(11)(12) to provide a framework for the analysis of panel rating data collected by using the Delphi method and related interactive rating schemes (13).…”

Modeling the Psychometric Properties of Complex Performance Assessment Tasks Using Confirmatory Factor Analysis: A Multistage Model for Calibrating Tasks

“…For example, the test specifications might call for a task of moderate difficulty at site S, with the constraint that the task was not previously exposure at site S. The parameters from the across-site, task-only model might be interpreted as preliminary estimates, to be used for initial task selection. These initial parameter estimates would be then updated using the within-site, taskrater model after the task was used for a period of time at site S. Parameter estimates for task-raters can be periodically evaluated to examine rater leniency and discrimination parameters over time, for example, to delineate rater drift (Wilson & Hoskens, 2001;Harik et al, 2009), to evaluate the accuracy of equating processes and to identify gaps in the task pool. Availability of rater generic and rater specific task and test statistics would help monitor rater performances and spot problematic raters.…”