Writing scale effects on raters: an exploratory study

Abstract

In writing assessment, finding a valid, reliable, and efficient scale is critical. Appropriate scales, increase rater reliability, and can also save time and money. This exploratory study compared the effects of a binary scale and an analytic scale across teacher raters and expert raters. The purpose of the study is to find out how different scale types impact rating performance and scores. The raters in this study rated twenty short EFL essays using the two scales, completed a rater cognition questionnaire, and took part in an in-depth interview. The ratings were analyzed using a multi-faceted Rasch analysis to compare essay scores and rater statistics across scales and rater groups. The results indicated when using the binary scale, the raters spent less time and were less spread out and more consistent in their ratings. Three out of four raters replied that less mental effort was required when using the binary scale and felt more confident in their ratings. Across the two rater groups, there was a bigger shift in rating performance when using the binary scale for the teacher raters than the expert raters. This implies that scale design had a greater effect on teacher raters. The overall findings suggest that the binary scale maybe a better fit for large scale assessment with sufficient rater training.

Raters’ perceptions of rating scales criteria and its effect on the process and outcome of their rating

“…According to the observations, raters felt more confusion in their ratings which implied that they got it quite demanding to clearly select one level. This finding is in line with Jeong (2019) who concluded that raters had to consider multiple areas in the analytic rating process and experienced more hesitation and rating conflict. Therefore, these strategies like anchoring happened more in analytic evaluations which can be due to the vague wording of the scale.…”

“…Thanks to this, it might be difficult for raters to analyze and interpret the scale components, and through adopting those strategies, they lessened the cognitive load. On the other hand, findings of Jeong ( 2019 ) showed that the scale design has a greater effect on the raters. Thus, designing rating scales and the rating criteria should provide an explicit and reliable foundation for scoring judgments, over and above distinguishing writing performance levels (Weigle, 2002 ).…”

“…The results illustrated that raters considered some features such as ideas and text structure more prominent than others. Jeong ( 2019 ) made a comparison between the effects of a binary scale and an analytic scale on rating performance and scores while using two groups of participants: teacher raters and expert raters. He found that because the binary scale lessened the rater cognitive load and was easy to use, the raters were more consistent in their ratings.…”

Triangulating natural language processing (NLP)-based analysis of rater comments and many-facet Rasch measurement (MFRM): An innovative approach to investigating raters’ application of rating scales in writing assessment

“…One examines rater scores quantitatively to detect rater effects; the other examines rater behavior qualitatively to explain how rater effects arise. Rating scores are typically analyzed by MFRM analysis (Jeong, 2019; Myford & Wolfe, 2003; Schaefer, 2008; Trace et al, 2017; Wang et al, 2017). Its central idea is to model many facets (e.g., examinee, criterion, and rater) in the rating process to predict the probability of a particular rating.…”

“…Whereas the former represents the product of scoring, the latter evidences the process of scoring (Greene et al, 1989). Rater scores are typically examined through many-facet Rasch measurement (MFRM) (e.g., Jeong, 2019; Myford & Wolfe, 2003; Schaefer, 2008), while rater behavior tends to be examined through surveys and/or interviews (e.g., Jeong, 2019), think-aloud protocols (e.g., Barkaoui, 2007), rater comments (e.g., H. J. Kim, 2015), or eye-tracking experiments (e.g., Ballard, 2017; Winke & Lim, 2015).…”