Judge Consistency and Severity Across Grading Periods

Abstract

The purpose of this research project was to confirm that differences in the severity of judges and the stringency of grading periods occur, regardless of the nature of the assessment or the examination materials used. Three rather different examinations that require judges were analyzed, using an extended Rasch model to determine whether differences in judge severity and grading-period stringency were observable for all three examinations. Significant variation in judge severity and some variation across grading periods were found on all three examinations. This implies that regardless of the nature of the examination, items, or judges, examinee/measures cannot be considered independent of the particular judges involved unless correction for severity is made systematically. Accounting for judge severity and gradinig-period stringency is extremely important when pass/fail decisions that are meant to generalize to competence are made, as in certification examinations.

Monitoring Faculty Consultant Performance in the Advanced Placement English Literature and Composition Program With a Many‐faceted Rasch Model

“…Some researchers contend that the level of severity a rater exercises is a relatively stable effect that changes little over time and is not modifiable by training (Bernardin and Pence, 1980;Lunz and Stahl, 1990;Lunz, Stahl, and Wright, 1996;O'Neill and Lunz, 1996;O'Neill and Lunz, 2000;Raymond, Webb, and Houston, 1991). By contrast, other researchers argue that some raters' levels of severity can shift substantially from reading to reading (Lumley and McNamara, 1993;Myford, Marr, and Linacre, 1996), from essay topic to essay topic (Bridgeman, Morgan, and Wang, 1996;Weigle, 1999), and from day to day within the same reading (Bleistein and Maneckshana, 1995;Braun, 1988;Coffman and Kurfman, 1968;Morgan, 1998;Wilson and Case, 2000;Wood and Wilson, 1974).…”

Monitoring Sources of Variability Within the Test of Spoken English Assessment System

“…Over the last several years, a number of performance assessment programs interested in examining and understanding sources of variability in their assessment systems have been experimenting with Linacre's (1999a) Facets computer program as a monitoring tool (see, for example, Heller, Sheingold, & Myford, 1998;Linacre, Engelhard, Tatum, & Myford, 1994;Lunz & Stahl, 1990;Myford & Mislevy, 1994;Paulukonis, Myford, & Heller, in press). In this study, we build on the pioneering efforts of researchers who are employing many-facet Rasch measurement to answer questions about complex rating systems for evaluating speaking and writing.…”

Investigating the Effect of Training on Raters’ Bias toward Test Takers in Oral Proficiency Assessment : A FACETS Analysis

“…The sixth column (z-scores, or standardized fit statistics) shows the test version rater bias estimate at this phase. Bias is the difference between expected and observed ratings of the obtained data, which is then divided by its standard error to derive the z-score (Lunz & Stahl, 1990). The most preferable z value is 0, which indicates that the data match the expected model, and thus, no rater bias.…”

“…Considerable evidence of poor rater consistency has been reported in some research (e.g., Lunz & Stahl, 1990;Trace, Janssen, & Meier, 2017), and even if adequate consistency might have been reported in most research, it is mostly on the basis of correlations alone. That is, even a perfect correlation might ignore systematic variations among raters.…”