Rater types in writing performance assessments: A classification approach to rater variability

Abstract

Research on rater effects in language performance assessments has provided ample evidence for a considerable degree of variability among raters. Building on this research, I advance the hypothesis that experienced raters fall into types or classes that are clearly distinguishable from one another with respect to the importance they attach to scoring criteria. To examine the rater type hypothesis, I asked 64 raters actively involved in scoring examinee writing performance on a large-scale assessment instrument to indicate on a four-point scale how much importance they would attach to each of nine routinely used criteria. The criteria covered various performance aspects, such as fluency, completeness, and grammatical correctness. In a preliminary step, many-facet Rasch analysis revealed that raters differed significantly in their views on the importance of the various criteria. A two-mode clustering technique yielded a joint classification of raters and criteria, with six rater types emerging from the analysis. Each of these types was characterized by a distinct scoring profile, indicating that raters were far from dividing their attention evenly among the set of criteria. Moreover, rater background variables were shown to partially account for the scoring profile differences. The findings have implications for assessing the quality of large-scale rater-mediated language testing, rater monitoring, and rater training.

VALIDATION OF AUTOMATED SCORES OF TOEFL IBT®TASKS AGAINST NONTEST INDICATORS OF WRITING ABILITY

“…One analysis that might shed light on some of the differences across topics would be a many-faceted Rasch analysis using the FACETS software (Linacre, 2010; see also Myford & Wolfe, 2003, 2004, for details of this method of analysis), which can be used to estimate rater severity and task difficulty on the same linear scale, allowing investigation of questions such as whether specific raters judge essays on certain topics more severely than others. This analysis could provide more detailed information about rater bias, and along with e-rater feature scores could complement recent research on the factors that influence rater behavior (e.g., Eckes, 2008).…”

“…One possible explanation for this result may be found in the research finding that essay raters do not base their scores strictly on the wording of a specific scale (see Eckes, 2008, for a recent review of the literature on rater behavior). For example, Lumley (2002) noted that raters' judgments seem to be based on "some complex and indefinable feeling about the text, rather than the scale content" and that raters form "a uniquely complex impression independently of the scale wordings."…”

Assessing EFL university students' writing: A study of score reliability

“…Raters also judge students' writing ability differently depending on their academic background and sex (Vann, Lorenz & Meyer, 1991), and the training received (Weigle, 1994). Studies such as Cumming (1990), Eckes (2008), Esfandiari & Myford (2013), González & Roux (2013), Lim (2011), Shi (2001), Shi, Wan, & Wen (2003) and Wiseman (2012), describe how distinct rater backgrounds influence (or do not influence) their rating behavior, actual scores and scoring procedures. Lim (2011), for instance, focused on experienced and inexperienced raters.…”

Az idegen nyelvi érettségi működése és hatása a tanulói teljesítmények és a tanári nézetek tükrében

“…Számos kutató (az íráskészség esetében például Weigle, 1998;Engelhard és Myford, 2003;Eckes, 2005Eckes, , 2008Schoonen, 2005; a beszédkészség esetében például Vidakovic és Galaczi, 2009) modellezte az értékelık tulajdonságait. E vizsgálatok tanulságai szerint az értékelık különbözhetnek az értéke-lési szempontok értelmezésében és alkalmazásában, a vizsgázók nyelvi teljesítményé-nek szigorú vagy enyhe megítélésében, az értékelési skálák megértésében és használa-tában, a különbözı készségszintő diákok értékelésének következetességében.…”