The Effects of Rater Severity and Rater Distribution on Examinees' Ability Estimation for Constructed‐response Items

Abstract

Since its 1947 founding, ETS has conducted and disseminated scientific research to support its products and services, and to advance the measurement and education fields. In keeping with these goals, ETS is committed to making its research freely available to the professional community and to the general public. Published accounts of ETS research, including papers in the ETS Research Report series, undergo a formal peer-review process by ETS staff to ensure that they meet established scientific and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that outside organizations may provide as part of their own publication processes. i AbstractThe current study used simulated data to investigate the properties of a newly proposed method (Yao's rater model) for modeling rater severity and its distribution under different conditions. Our study examined the effects of rater severity, distributions of rater severity, the difference between item response theory (IRT) models with rater effect and without rater effect, and the difference between the precision of the ability estimates for tests composed of only constructed-response (CR) items and for tests composed of multiple-choice (MC) and CR items combined. Our results indicate that rater severity and its distribution can increase the bias of examinees' ability estimates and lower test reliability. Moreover, using an IRT model with rater effects can substantially increase the precision in the examinees' ability estimates, especially when the test was composed of only CR items. We also compared Yao's rater model with Muraki's rater effect model (1993) in terms of ability estimation accuracy and rater parameter recovery. The estimation results from Yao's rater model using Markov chain Monte Carlo (MCMC) were better than those from Muraki's rater effect model using marginal maximum likelihood.

Monitoring of Scoring Using the e‐rater® Automated Scoring System and Human Raters on a Writing Test

“…Unfortunately, the literature on effective quality control procedures using quality control tools on automated scores or long‐term monitoring on both human and automated scores is sparse. However, many studies have been conducted on human scoring and rater effects (DeCarlo, ; Donoghue, McClellan, & Gladkova, ; Engelhard, , ; Longford, ; Myford & Wolfe, ; Patz, Junker, Johnson, & Mariano, ; Wang & Yao, ; Wilson & Hoskens, ; Wolfe & Myford, ). The results from these studies indicate that biases of examinee ability estimates or systematic error may be caused by varying degrees of rater leniency or central tendency.…”

“…The results from these studies indicate that biases of examinee ability estimates or systematic error may be caused by varying degrees of rater leniency or central tendency. Additionally, rater effects can increase these bias estimates and lower test reliability (Donoghue et al, ; Wang & Yao, ).…”

“…An issue exists regarding how human raters are assigned to essays (Wang, ). Wang and Yao () used simulated data to examine human‐rater effects. The results of their study indicated that the way essays are assigned to human raters, and the human‐rater effects, especially the effect of human‐raters' scoring severity, can increase the student ability estimate bias and lower test reliability.…”

“…Few research articles (Lee & von Davier, ; Luecht, ) have proposed the use of quality control techniques to monitor the scoring, equating, and reporting of test scores. The rater effects may substantially increase the bias in students' final scores without careful monitoring (Wang & Yao, ). Walker (), Wang (), and Wang and von Davier () proposed a set of statistics and a framework (examinee, test, prompt level, and rater level) that can be used to monitor the quality and consistency of CR scoring.…”

Detecting rater effects in trend scoring

“…A number of models exist that take into account rater effects. Examples include the many-faceted Rasch model (Linacre, 1989); FACETS model (Lunz, Wright , & Linacre, 1990); an IRT model for multiple raters (Verhelst & Verstralen, 2001); the rater bundle model ; the hierarchical rater model (Patz, Junker, Johnson, & Mariano, 2002) and its signal detection theory version (DeCarlo, 2010;DeCarlo, Kim, and Johnson, 2011); and Yao's rater model (Wang & Yao, 2013). These models are most useful when all the CR items of an assessment have been scored and merged with the multiple choice items (Sgammato & Donoghue, 2018).…”