A Comparison of Paper and Online Tests Using a Within-Subjects Design and Propensity Score Matching Study

Abstract

This inquiry had 2 components: (1) the first was substantive and focused on the comparability of paper-based and computer-based test forms and (2) the second was a within-study comparison wherein a quasi-experimental method, propensity score matching, was compared with a credible benchmark method, a within-subjects design. The tests used in the comparison of online tests and paper-based tests were End-of-Course tests in Algebra and English, in a statewide high school testing program. Students were tested in Grades 8 and 9. In general, the substantive studies suggested that the online and paper tests appeared to be measuring the same underlying constructs with the same level of reliability. The within-study portion of the investigation indicated that propensity score matching study yielded results that were virtually identical to the outcome of the more conventional within-subjects experimental design. Both the methodological and substantive aspects of this investigation yielded outcomes that should be of interest to investigators in both of these areas.

Does Matching Quality Matter in Mode Comparison Studies?

“…Consistent with what can be expected from the covariates balance results, the two propensity score matching approaches (i.e., the Lottridge et al, 2011, and the OPM-OBT approach) are most alike in classifying students based on the raw scores associated with the proficiency cuts. However, we also observed that the scale scores at the proficiency cuts are most alike for the modified MSCA and the Lottridge et al (2011) approach. We thus inferred that a different nonlinear relationship was established between raw scores and scale scores for the three matched PPT data sets.…”

“…Consequently, some statistical methods have been proposed to obtain comparable PPT and CBT samples without the need for students to take a test in both modes. Two approaches are matched samples comparability analyses (MSCA; Way, Davis, & Fitzpatrick, 2006;Way et al, 2008) and propensity score matching (e.g., Lottridge et al, 2011). Propensity score matching and the MSCA method are similar in attempting to create statistically matched samples (Way et al, 2008), though the latter seems to match only on previous achievement scores (Way et al, 2006(Way et al, , 2008.…”

“…Propensity score matching and the MSCA method are similar in attempting to create statistically matched samples (Way et al, 2008), though the latter seems to match only on previous achievement scores (Way et al, 2006(Way et al, , 2008. A modified MSCA method was considered in this study, as well as the propensity score matching approach used by Lottridge et al (2011). Details of these two approaches and a different propensity score matching approach delineated here are provided below.…”

“…We hoped that including those variables as covariates would help to remove some bias with regard to different computer-person interaction experience due to culture differences. The only possible important variable we did not include (based on comparing with mode comparison literature such as Bennett et al [2008] and Lottridge et al [2011]) was student computer skill scores, as this information has not been collected at the state level in the state from which the data came.…”

Examining the effects of paper-based and computer-based modes of assessment on mathematics curriculum-based measurement

“…In order to compare End-of-Course assessments in Algebra I and English I, Lottridge, Nicewander, and Mitzel (2011) had participants complete PBT and CBT versions of a multiple-choice assessment created by the state. Computer skills, reading scores, math scores, areas of the state, free and reduced lunch status, special education/gifted status, and ethnicity were all included in the analysis.…”