Evaluating CEFR rater performance through the analysis of spoken learner corpora

Abstract

Background: Although teachers of English are required to assess students' speaking proficiency in the Common European Framework of Reference for Languages (CEFR), their ability to rate is seldom evaluated. The application of descriptors in the assessment of English speaking on CEFR in the context of English as a foreign language has not often been investigated, either. Methods: The present study first introduced a form of rater standardization training. Two trained raters then assessed the speaking proficiency of 100 learners by means of actual corpus data. The study then compared their rating results to evaluate interrater reliability. Next, ten samples of exact/adjacent agreement between Raters 1 and 2 were rated by six teachers of English in tertiary education. Two of them had attended rater standardization training with Raters 1 and 2, while the other four had not received any relevant training. Results: The two raters agreed exactly in 44% of cases. The rating results between the two trained raters were closely correlated (ρ = .893). Cross-tabulation showed that in one third of the samples, Rater 2 scored higher than Rater 1 and they agreed more often at the higher levels. The better rating performance of Teachers 1 and 2 suggested that rater standardization training may have helped enhance their performance. The unsatisfactory proportion of correctly assigned levels in teachers' ratings overall was probably due to the high input of subjective judgment based on vague CEFR descriptors. Conclusions: Regarding assessment, it is shown that the attendance of rater standardization training is of help in assessing learners' speaking proficiency in CEFR. This study provides a model for assessing data from spoken learner corpora, which adds an important attribute to future studies of learner corpora. The paper also raises doubts about teachers' ability to evaluate students' speaking proficiency in CEFR. As CEFR has been widely adopted in the relevant fields of English language teaching and assessment, it is suggested that the rating training framework established in this study, which uses learner corpus data, be offered to (prospective) teachers of English in tertiary education.

Rating Performance among Raters of Different Experience Through Multi-Facet Rasch Measurement (MFRM) Model

“…It signifies that the severity practiced by the two groups was not identical. This finding is consistent with those of Attali (2016); Davis (2016) and Huang et al (2018), who reported that raters with varying rating experience provided heterogeneous ratings, even though the studies were implemented in different contexts. This consistency may be due to how rating experience among raters was operationally defined.…”

“…Empirically, conflicting findings emerged from the literature in terms of how raters' experience has impacted rating quality. Raters of different experiences were reported to show distinct rating quality in some studies (Davis 2016;Huang et al 2018;Kim 2015), but differences were not observed in other studies (Ahmadi Shirazi, 2019;Isaacs & Thomson, 2013;Şahan & Razı, 2020).…”

“…However, both groups of raters were successful in generating homogeneous ratings after training. Huang et al (2018) found out that raters showed different levels of severity when they are compared according to their first language within the context of language testing. Recently, Ahmadi Shirazi (2019) assigned raters of writing test to rate using two rating methods (holistic and analytical) and concluded that raters of writing test displayed different levels of severity and leniency.…”

The Rater Performance Categorization System (RPCS) for Intensive English Programs

“…It is not a common practice to evaluate and monitor the teacher's or rater's performance in IEPs (Huang, et al, 2018). It is thought by the author of this paper that there are some reasons behind it.…”

Brunswik’s Lens Model: This Is How to Inspire Accurate Raters

“…It was learned that raters' severity remained consistent before and after the training yet level of agreement among raters improved at the end of the scoring sessions. Bijani (2018) and Huang, Kubelec, Keng, & Hsu (2018) have chosen samples from experienced raters and inexperienced raters. Both studies concluded that the two groups of raters were able to attain the same standard of inter-rater reliability after rater training was given.…”