The affectability of writing assessment scores: a G-theory analysis of rater, task, and scoring method contribution

Abstract

The present study attempted to to investigate factors which affect EFL writing scores through using generalizability theory (G-theory). To this purpose, one hundred and twenty students participated in one independent and one integrated writing tasks. Proceeding, their performances were scored by six raters: one self-rating, three peers,-rating and two instructors-rating. The main purpose of the sudy was to determine the relative and absolute contributions of different facets such as student, rater, task, method of scoring, and background of education to the validity of writing assessment scores. The results indicated three major sources of variance: (a) the student by task by method of scoring (nested in background of education) interaction (STM:B) with 31.8% contribution to the total variance, (b) the student by rater by task by method of scoring (nested in background of education) interaction (SRTM:B) with 26.5% of contribution to the total variance, and (c) the student by rater by method of scoring (nested in background of education) interaction (SRM:B) with 17.6% of the contribution. With regard to the G-coefficients in G-study (relative G-coefficient ≥ 0.86), it was also found that the result of the assessment was highly valid and reliable. The sources of error variance were detected as the student by rater (nested in background of education) (SR:B) and rater by background of education with 99.2% and 0.8% contribution to the error variance, respectively. Additionally, ten separate G-studies were conducted to investigate the contribution of different facets across rater, task, and methods of scoring as differentiation facet. These studies suggested that peer rating, analytical scoring method, and integrated writing tasks were the most reliable and generalizable designs of the writing assessments. Finally, five decision-making studies (D-studies) in optimization level were conducted and it was indicated that at least four raters (with G-coefficient = 0.80) are necessary for a valid and reliable assessment. Based on these results, to achieve the greatest gain in generalizability, teachers should have their students take two writing assessments and their performance should be rated on at least two scoring methods by at least four raters.

Test review of Iranian English language proficiency test: MSRT test

“…The answer to such a question could be found in classical test theories but in generalizability analysis of the test results. Usually in the administration of high-stakes and standardized tests the test administration context, the number of items, and the same scoring procedure have kept stable in order to eliminate any source of construct irrelevant variance (Khodi, 2021). This could be problematic due to two reasons: first, there is no intention to revise and improve the test quality by the test administrators and developers; second, keeping these conditions stable without examining the relative and absolute contribution they have would not be really insightful and helpful in reliability assessment.…”

Experienced and Novice L2 Raters’ Cognitive Processes while Rating Integrated and Independent Writing Tasks

“…The process of rating writing tasks may also be affected by other factors such as the type of task or the scoring method. Recently, Khodi (2021) in a G-theory analysis of rater, task, and scoring method examined the affectability of writing assessment scores. Using various raters, tasks, and scoring methods, he found that to reach maximum generalizability, students should take two writing tasks and their performance should be evaluated by at least four raters using at least two scoring methods.…”

Raters’ perceptions of rating scales criteria and its effect on the process and outcome of their rating

“…The characteristics of raters in writing assessment are the idiosyncrasy that can be sought in the cognitive, attitudinal, and background variables. These factors are affected by raters’ appropriate education, training, and prior experience (Khodi, 2021 ; Kim & Lee, 2015 ; Rahayu, 2020 ). To prevent assigning inconsistent scores, concentrating properly and exclusively on evaluation of examinee’ writing performance during the rating process, and applying appropriate rating procedures, raters should receive professional training.…”