Investigating the Impact of Rater Training on Rater Errors in the Process of Assessing Writing Skill

Abstract

In the process of measuring and assessing high-level cognitive skills, interference of rater errors in measurements brings about a constant concern and low objectivity. The main purpose of this study was to investigate the impact of rater training on rater errors in the process of assessing individual performance. The study was conducted with a pretest-posttest control group quasi-experimental design. In this research, 45 raters were employed, 23 from the control group and 22 from the experimental group. As data collection tools, a writing task that was developed by IELTS and an analytical rubric that was developed to assess academic writing skills were used. As part of the experimental procedure, rater training was provided and this training was implemented by combining rater error training and frame of reference training. When the findings of the study were examined, it was found that the control and experimental groups were similar to each other before the experiment, however, after the experimental process, the study group made more valid and reliable measurements. As a result, it was investigated that the rater training given had an impact on rater errors such as rater severity, rater leniency, central tendency, and Halo effect. Based on the obtained findings, some suggestions were offered for researchers and future studies.

The Role of Time on Performance Assessment (Self, Peer and Teacher) in Higher Education: Rater Drift

“…When the literature is examined, it is seen that numerous studies tend to depict rater effects as static properties of raters (i.e., as if a rater effect affects the performance of each student in precisely the same way) in the self-, peer-, and teacher assessments of student performance (Engelhard & Myford, 2003;Harik et al, 2009;Hoskens & Wilson, 2001;Lamprianou, 2006;Leckie & Baird, 2011;McLaughlin et al, 2009;Sata & Karakaya, 2022;Uto, 2022). However, the behavior of an individual rater can change over time, and differential rater functioning over time (drift) can express this change in scorer performance over time.…”