Validating Translation Test Items via the Many-Facet Rasch Model

Abstract

This study applied the many-facet Rasch model to assess learners' translation ability in an English as a foreign language context. Few attempts have been made in extant research to detect and calibrate rater severity in the domain of translation testing. To fill the research gap, this study documented the process of validating a test of Chinese-to-English sentence translation and modeled raters' scoring propensity defined by harshness or leniency, expert/novice effects on severity, and concomitant effects on item difficulty. Two hundred twenty-five, third-year senior high school Taiwanese students and six educators from tertiary and secondary educational institutions served as participants. The students' mean age was 17.80 years ( SD = 1.20, range 17-19). The exam consisted of 10 translation items adapted from two entrance exam tests. The results showed that this subjectively scored performance assessment exhibited robust unidimensionality, thus reliably measuring translation ability free from unmodeled disturbances. Furthermore, discrepancies in ratings between novice and expert raters were also identified and modeled by the many-facet Rasch model. The implications for applying the many-facet Rasch model in translation tests at the tertiary level were discussed.

Evaluating students’ ability in constructing scientific explanations on chemical phenomena

“…Uni-dimensionality is a measure to ensure that the developed test instrument is able to measure the construct, meaning that the item only measures one construct at a time, namely scientific argumentation skills (Bond & Fox, 2007). Unidimensionality measurement, in the present work, uses the principal component analysis (PCA) of residuals to estimate the extent to which instrument diversity measures what it is supposed to measure (Aryadoust et al, 2021;Ding, 2018;Sumintono & Widhiarso, 2014;Tseng et al, 2019). If the measurement result shows data that closely fit the Rasch model, most of the non-random variance (not randomized) found in the data can be explained by one latent dimension (Chi et al, 2021;Eckes, 2015).…”

<p>The Reliability and Validity of the “Activity and Participation” Component in the Brief ICF Core Set for Chronic Obstructive Pulmonary Diseases Based on Rasch Analysis</p>

“…23 MFR can distinguish and isolate the interaction of each facet when measuring each facet, which can greatly reduce the influence of the rater's subjectivity and the patient's own ability on various categories. 24,25 Therefore, using the Rasch model for the reliability and validity tests of the ICF core set was helpful to provide more detailed information about ICF from various aspects such as raters, subjects, and categories. The Rasch model has been widely used in many fields.…”