Effects of training on raters of ESL compositions

Abstract

Several effects of training on composition raters have been hypothesized but not investigated empirically. This article presents an analysis of the verbal protocols of four inexperienced raters of ESL placement compositions scoring the same essays both before and after rater training. The verbal protocols show that training clarified the intended scoring criteria for raters, modified their expectations of student writing and provided a reference group of other raters with which raters could compare themselves, although agreement with peers was not an over-riding concern. These results are generally in accordance with hypothesized effects of rater training.

Assessing EFL university students' writing: A study of score reliability

“…Whether assessment practice is part of training or not is a crucial issue in training teachers to assess writing. By providing teachers with the opportunity to practice and use the assessment tool that the school or institution encourages, assessment reliability may be improved (Weigle, 1994). These results are also in line with those found by Wiseman (2012), who concluded that rater background had an impact on raters' use of scoring criteria.…”

“…The results of this study allowed the researchers to conclude that the use of assessment tools, such as scoring rubrics, is not enough to improve rater reliability (González & Roux, 2013;Saxton et al 2012;Weigle, 1994), and therefore these results may have important implications for teaching practice. First, it is important to consider that rubrics are tools that teachers may use to facilitate their assessment of writing.…”

“…For instance, experts have found that raters respond to different aspects of writing, and they do so with some internal inconsistency depending on, for example, their experiential background and their views on the students' linguistic and rhetorical backgrounds (Hamp-Lyons, 1989). Raters also judge students' writing ability differently depending on their academic background and sex (Vann, Lorenz & Meyer, 1991), and the training received (Weigle, 1994). Studies such as Cumming (1990), Eckes (2008), Esfandiari & Myford (2013), González & Roux (2013), Lim (2011), Shi (2001), Shi, Wan, & Wen (2003) and Wiseman (2012), describe how distinct rater backgrounds influence (or do not influence) their rating behavior, actual scores and scoring procedures.…”

Construct Under-representation and Construct Irrelevant Variances on IELTS Academic Writing Task 1: Is There Any Threat to Validity?

“…Scoring rubrics sheet called performance descriptors sheet typical of analytical rating, extracted from IELTS Handbook (2007) were given to both raters and examinees. It was submitted to the raters in order mainly to avoid rater's bias (Henning, 1987;Weigle, 1999;Eckes, 2012;Ecks, 2008) and to free them from evaluative personal judgment (Weigle, 1994(Weigle, , 1998) and background variables (Johnson & Lim, 2009) potentially affecting their scoring process. In the case of the examinees, the objective was to engage them in the related prompts (Bachman & Palmer, 2000) and to keep them cognizant of the underlying construct to be tested.…”

On Rater Agreement and Rater Training

“…Thus, the rater, as opposed to the test materials, candidates or rating scale is used as the window through which the evaluation of second language speaking performance can be observed. This type of qualitative data can tell us about the process of discriminating between candidates (Pollitt and Murray, 1996), and is useful for the purpose of rater training (Weigle, 1994), and for investigating how raters reach their decisions (Milanovic et al, 1996;Taylor, 2000). The latest research of the type in China is a study on the interpretation and application of the assessing criteria in TEM4-Oral (Wang Haizhen, 2008).…”