Individual Feedback to Enhance Rater Training: Does It Work?

Elder, Catherine; Knoch, Ute; Barkhuizen, Gary; Randow, Janet von

doi:10.1207/s15434311laq0203_1

Cited by 66 publications

(56 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fourth, rating context; which includes the medium and process through which responses are distributed to raters, rater training procedures, rater monitoring and feedback practices, and temporal and physical features of the rating environment; also influences the quality of ratings. Several studies have supported the notion that rater training increases the quality of ratings (Shohamy et al, 1992;Sweedler-Brown, 1985) and that the self-pacing that is afforded by an online training module increases the efficiency of that training process (Wolfe, Matthews, & Vickers, 2010;Elder, Barkhuizen, Knoch, & von Randow, 2007;Elder, Knoch, Barkhuizen, & von Randow, 2005;Knoch, Read, & von Randow, 2007). Less is known about the contribution to rating quality of response distribution systems, rater monitoring and feedback processes, and the setting within which ratings are assigned.…”

Section: Influences On Rater Agreement and Accuracymentioning

confidence: 99%

Features of difficult-to-score essays

2016

View full text Add to dashboard Cite

Section: Influences On Rater Agreement and Accuracymentioning

confidence: 99%

Features of difficult-to-score essays

2016

View full text Add to dashboard Cite

“…Raters' reactions to the program were also mixed, some being positive with others offering suggestions for improving the program or expressing preference for the face-to-face training. Elder et al (2005) examined the effects of individual rater performance feedback on the scoring of the same writing test employed in Elder et al (2007). The eight raters began by scoring 100 Diagnostic English Language Needs Assessment (DELNA) scripts, received online training, and then scored 50 randomly sampled DELNA writing scripts.…”

Section: Effect Of Rater Training On Scoring Performancementioning

confidence: 99%

“…Three of the five studies reviewed above explored the effects of a conventional training method or procedure, with the exception of Elder et al (2005) and Knoch et al (2007), which looked into the use of customized rater feedback. Weigle (1998) used a pre-post-training design with a single rater group, and Shohamy et al (1992) employed a study design that involved an experimental group and a control group.…”

Section: Effect Of Rater Training On Scoring Performancementioning

confidence: 99%

How Do Raters From India Perform in Scoring the Toefl Ibt™ Speaking Section and What Kind of Training Helps?

Mollaun

2009

ETS Research Report Series

View full text Add to dashboard Cite

This study investigated the scoring of the Test of English as a Foreign Language TM Internet-based Test (TOEFL iBT™) Speaking section by bilingual or multilingual speakers of English and 1 or more Indian languages. We explored the extent to which raters from India, after being trained and certified, were able to score the Speaking section for TOEFL iBT examinees with mixed first language (L1) backgrounds, especially those speaking an Indian language, accurately and consistently. The effectiveness of a special training package designed for scoring Indian examinees was examined as well. A total of 26 trained and certified raters from India were randomly divided into 2 groups and participated in 2 on-site scoring sessions in Mumbai. In the first session, both groups received regular training for scoring the TOEFL iBT Speaking section, which was largely similar to that received by raters in North America. In the second scoring session, 1 group continued to receive the regular training while the second group was trained using a special training package. Rater feedback surveys were also given to the raters. It was found that with training similar to that which operational U.S.-based raters receive, the raters from India performed as well as the operational raters in scoring both Indian and non-Indian examinees. In addition, the special training helped the raters score Indian examinees more consistently, leading to increased score reliability estimates. It also boosted raters' levels of confidence in scoring Indian examinees.Key words: TOEFL iBT, speaking, speech scoring, rater background characteristics, non-native raters, rater biasii To obtain more information about the TOEFL programs and services, use one of the following:

show abstract

“…Others have written about or investigated what factors do (and do not) contribute to reliable marking -for example, the use of exemplar work (Sadler 1987;Wolf 1995;Baird, Greatorex, and Bell 2004), discussion between assessors (Black et al 1989;Wolf 1995), discussion and consensus between *Corresponding author. Email: greatorex.j@cambridgeassessment.org.uk Downloaded by [Florida State University] at 14:19 26 December 2014 examiners (Baird, Greatorex, and Bell 2004), feedback on marking performance (Wigglesworth 1993;Shaw 2002;Elder et al 2005), training (Lunz, Wright, and Linacre 1990;Shohamy, Gordon, and Kraemer 1992), experience (Ruth and Murphy 1988;Weigle 1999;Shohamy, Gordon, and Kraemer 1992), contrast with previous candidates' performances (Spear 1997), which item or question an examiner is marking (Wiseman and Wrigley 1958;Black 1962) and examiners' personality characteristics (Branthwaite, Trueman, and Berrisford 1981;Greatorex and Bell 2004). This list is not intended to be exhaustive.…”

Section: Introductionmentioning

confidence: 99%

“…This list is not intended to be exhaustive. Some of this literature relates to GCE or GCSE marking (for example, Greatorex and Bell 2004) but some is about marking in other contexts -for example, university writing assessment programmes (Elder et al 2005). Although there is a great deal of research about the reliability of marking there are still some remaining research questions related to the GCSE and GCE context.…”

Section: Introductionmentioning

confidence: 99%

What makes AS marking reliable? An experiment with some stages from the standardisation process

Greatorex

Bell

2008

Research Papers in Education

View full text Add to dashboard Cite

It is particularly important that GCSE and A-level marking is valid and reliable as it affects the life chances of many young people in England. Current developments in marking technology are coinciding with potential changes in procedures to ensure valid and reliable marking. In this research the effectiveness of procedures to facilitate the reliability of marking were evaluated using a design with three interventions. Two of the interventions, the standardisation meeting and personal feedback, are similar to the conventional approach in GCSE and A-level practice but within the constraints of a research study. The other intervention, pre-written feedback, is how examples of marked scripts might be presented for part of examiner standardisation or training in other situations; it is also similar to the approach to feedback used in earlier research by Shaw and Baird et al. Each participating examiner experienced two of the three types of intervention. Marking data were gathered before and after the first intervention as well as after the second intervention. Inter-examiner reliability was analysed using ANOVA models. Due to the different levels of inter-examiner reliability at the beginning of the experiment it was hard to compare the effectiveness of the different interventions. Nevertheless, this research highlights that: (1) in line with previous research, standardisation meetings on their own are not particularly effective for experienced examiners;(2) as has been found in previous research, more experienced examiners were more lenient; (3) most combinations of interventions (combinations of standardisation processes -including the conventional approach) led to a decline in the size of the marking error; and (4) sometimes feedback (personal or pre-written) was effective and sometimes it was not.

show abstract

Individual Feedback to Enhance Rater Training: Does It Work?

Cited by 66 publications

References 11 publications

Features of difficult-to-score essays

Features of difficult-to-score essays

How Do Raters From India Perform in Scoring the Toefl Ibt™ Speaking Section and What Kind of Training Helps?

What makes AS marking reliable? An experiment with some stages from the standardisation process

Contact Info

Product

Resources

About