Establishing an Operational Model of Rating Scale Construction for English Writing Assessment

Abstract

Rating scales for writing assessment are critical in that they determine directly the quality and fairness of such performance tests. However, in many EFL contexts, rating scales are made, to certain extent, based on the intuition of teachers who strongly need a feasible and scientific route to guide their construction of rating scales. This study aims to design an operational model of rating scale construction with English summary writing as an example. Altogether 325 university English teachers, 4 experts in language assessment and 60 English majors in China participated in the study. 20 textual attributes were extracted, through text analysis, from China’s Standards of English Language Ability (CSE), theoretical construct of summary writing, comments on sample summary writing essays from 8 English teachers and their personal judgement. The textual attributes were then investigated through a large-scale questionnaire survey. Exploratory factor analysis and expert judgement were employed to determine rating scale dimensions. Regression analysis and expert judgement were conducted to determine the weighting distribution across all dimensions. Based on such endeavors, a tentative operational model of rating scale construction was established, which can also be applied and adapted to develop rating scales in other writing assessment. 

The Literature Review of Writing Test in Past Decade in China

“…On the contrary, assessment approach and genres of writing with high scoring validity have been less studied in the testing field. In fact, the highly subjective nature of writing test scoring has forced scholars to grapple with how to make writing tests a fairer and more equitable test using a consistent and specific scale, on the other hand, this issue has been difficult to achieve compared to other test types [10][11][12][13][14][15][16]. As a type of language test with high stakes and large-scale characteristics in China, the writing of CET-4 and CET-6 have been the focus of research in the field of writing test.…”