Does Matching Quality Matter in Mode Comparison Studies?

Abstract

This article provides a brief overview and comparison of three matching approaches in forming comparable groups for a study comparing test administration modes (i.e., computer-based tests [CBT] and paper-and-pencil tests [PPT]): (a) a propensity score matching approach proposed in this article, (b) the propensity score matching approach used by Lottridge, Nicewander, and Mitzel, and (c) a modified approach of matched samples comparability analyses (MSCA) mentioned by Way, Davis, and Fitzpatrick. Different matching approaches resulted in different matched data with differing degrees of matching quality, and matched data from each matching approach were then used in the mode comparison investigation. Construct equivalence was examined and the level of invariance was found to be consistent across modes for all three matching approaches. Raw-to-scale score conversion tables were created, and the impact on CBT students' proficiency classification was examined. The comparison of the number of CBT students whose proficiency classification would be affected and the equality of score distributions between modes on raw scores and scale scores across the three matching approaches indicate that the propensity score matching approach delineated in this article led to the most consistent evidence for the conclusion of the mode comparison.

Mode Effects in College Admissions Testing and Differential Speededness as a Possible Explanation

“…Results from six studies favored paper testing in the following areas of language arts: 9th‐grade English (Kim & Huynh, 2010; Lottridge et al., 2011), reading comprehension (Jerrim, 2016; Kim & Kim, 2013; Mangen et al., 2013), and English language arts/literacy (Liu et al., 2016). Only one study indicated comparability on English language proficiency tests in some analyses for students in grades 9–12 (Zeng et al., 2015). In studies of high school mathematics subjects such as algebra and geometry, four studies had results that favored paper testing (Jerrim, 2016; Liu et al., 2016; Lottridge et al., 2011; Minnesota Department of Education, 2016) and two indicated comparability (Liu et al., 2016; Moon, 2013).…”

Two Statistics for Measuring the Score Comparability of Computerized Adaptive Tests

“…For example, if test forms were available in paper and computer-based versions, one may compare the means for paper and computer-based tests to see if they were comparable. Many score comparability studies of this type exist (see Wang et al, 2007Wang et al, , 2008Zeng et al, 2015). In the context of evaluating score comparability of alternate CAT forms, comparing observed score distributions or moments of these distributions is challenging because few examinees typically take each form and performance on alternate forms is expected to differ as a function of ability.…”