Self-evaluation of rater bias in written composition assessment

Abstract

No assessment is entirely free of bias. This paper presents findings concerning the way raters in the research group evaluate the extent to which they are influenced by various types of rater bias when grading their students’ written compositions. The sources of bias covered in the article include the teacher’s knowing the student writer and his or her proficiency in English, the difficulty of the writing task, distressful content likely to trigger the rater’s emotional reaction, the test taker’s views clashing with those of the rater, students’ progress, and the like. The data were gathered by the participants in the study via a questionnaire. In addition, the researcher’s interpretation of the respondents’ answers was verified through interviews. Although the two research methods and self-evaluation have their drawbacks, the results reveal interesting, relevant and important information on aspects which make written composition assessment less reliable and valid. The findings confirm the need to raise raters’ awareness of the causes of bias to which they are most susceptible, bringing them closer to effectively addressing the problem of assessment bias. The research involving eleven lecturers teaching Language in Use at the Department of English and American Studies at the Faculty of Arts, University of Ljubljana, is a part of a much larger project based on the author’s PhD thesis.

Raters’ Bias, Background and Perception in Awarding Score of Writing Performance

“…I propose that the raters' background, perception of the test construct, perception of the scoring rubric and using the same rubric would be sufficient to confirm the rater's reliability, if I combine all the components of raters' background, perception of the scoring rubric, using the same scoring rubric and statistical analysis of numerical data. (Sokolov, 2014) I assume that the result can give relevant information on the education setting in which the teacher or raters operate. (Koretz, 2008) Therefore in this study, the combination of raters' background, perception of the scoring rubric, using the same scoring rubric and statistical analysis of numerical data were applied together.…”