Comparison of Classical Test Theory vs. Multi-Facet Rasch Theory in writing assessment

Abstract

Testing English writing skills could be multi-dimensional; thus, the study aimed to compare students' writing scores calculated according to Classical Test Theory (CTT) and Multi-Facet Rasch Model (MFRM). The research was carried out in 2019 with 100 university students studying at a foreign language preparatory class and four experienced instructors who participated in the study as raters. Data of the study were collected by using a writing rubric consisting of four components (content, organization, grammar and vocabulary). Participants' writing scores were analysed thoroughly both by CTT and MFRM. At the first step, the participants' writing scores were calculated by taking the means of the writing points given by the graders in the CTT model. Then, the MFRM was applied to the data through a three-facet design considering the rater, student and rubric components as MFRM facets respectively. Finally, ability estimates obtained and reported in the logit scale via Rasch Analysis were converted into the analytic rubric's component scores used throughout the scoring procedure. Finally, two sets of writing scores were calculated and compared according to both measurement models. Considering the findings, it was summoned that there was a positive and high correlation between the ability estimates found according to the CTT and the MFRM. However, the mean score difference calculated according to both theories was still significant. Moreover, the analyses showed that criterion validity of the writing scores obtained via the MFRM was higher than the scores obtained via the CTT.

Exploring the Growth and Predictors of Fine Motor Skills in Young Children Aged 4–8 Years

“…Furthermore, the mean measure of the z-standardized measures (ZSTD outfit) for both instruments were 0.10 for students and 0.30 for items, which align with the recommended values of −2 to +2, as stated by Andrich [60]. The Chi-square values and degrees of freedom (χ 2 /df < 3) for the instrument indicated that the analyzed data followed a normal distribution in the Rasch model [61]. Table 2 demonstrates that both item and person reliabilities were found to be satisfactory.…”

Exploring the Accuracy and Consistency of a School Readiness Assessment Tool for Preschoolers: Reliability, Validity and Measurement Invariance Analysis

“…However, CTT does not account for item difficulty or variability in individual differences in ability levels (Ayanwale et al 2022) and MI testing (Siregar and Panjaitan 2022). IRT is a modern approach to psychometric measurement that models the relationship between a person's ability level and their responses to test items (Polat et al 2022). IRT assumes that items have varying degrees of difficulty and discrimination, allowing the estimation of individuals' abilities based on their responses (Liu et al 2022).…”

“…It is crucial to acknowledge that these instruments need to undergo psychometric evaluation to ensure their suitability for different participants and varying timeframes (Liu et al 2020). Additionally, emphasis should be placed on incorporating assessment theories during the development of psychological scales (Polat et al 2022).…”

Diagnostic Assessment of School Readiness

“…However, CTT does not account for item difficulty or variability in individual differences in ability levels (Ayanwale et al, 2022) and MI testing (Siregar & Panjaitan, 2022). IRT is a modern approach to psychometric measurement that models the relationship between a person's ability level and their responses to test items (Polat et al, 2022). IRT assumes that items have varying degrees of difficulty and discrimination, allowing the estimation of individuals' abilities based on their responses (Liu et al, 2022).…”