The TIMSS 2019 Item Equivalence Study: examining mode effects for computer-based assessment and implications for measuring trends

Abstract

Background: TIMSS 2019 is the first assessment in the TIMSS transition to a computerbased assessment system, called eTIMSS. The TIMSS 2019 Item Equivalence Study was conducted in advance of the field test in 2017 to examine the potential for mode effects on the psychometric behavior of the TIMSS mathematics and science trend items induced by the change to computer-based administration. Methods: The study employed a counterbalanced, within-subjects design to investigate the potential for eTIMSS mode effects. Sample sizes for analysis included 16,894 fourth grade students from 24 countries and 9,164 eighth grade students from 11 countries. Following a review of the differences of the trend items in paper and digital formats, item statistics were examined item by item and aggregated by subject for paperTIMSS and eTIMSS. Then, the TIMSS scaling methods were applied to produce achievement scale scores for each mode. These were used to estimate the expected magnitude of the mode effects on student achievement. Results: The results of the study provide support that the mathematics and science constructs assessed by the trend items were mostly unaffected in the transition to eTIMSS at both grades. However, there was an overall mode effect, where items were more difficult for students in digital formats compared to paper. The effect was larger in mathematics than science. Conclusions: Because the trend items cannot be expected to be sufficiently equivalent across paperTIMSS and eTIMSS, it was concluded that modifications must be made to the usual item calibration model for TIMSS 2019 to measure trends. Each eTIMSS 2019 trend country will administer paper trend booklets to a nationally representative sample of students, in addition to the usual student sample, to provide a bridge between paperTIMSS and eTIMSS results.

Tablet assessment in primary education: Are there performance differences between TIMSS' paper‐and‐pencil test and tablet test among Dutch grade‐four students?

“…Students can be attracted by the use of a digital device for their assessment because of the novelty of the device or because they enjoy using the device in their daily life (Jerrim, 2016). Both computers and tablets present students with engaging, interactive items, such as interactive simulations (Fishbein, Martin, Mullis, & Foy, 2018). Furthermore, the use of tablets might be more complicated when reading on screen or when the use of scrap paper is required; at the same time, however, the touchscreen feature facilitates many actions (such as navigating), especially when students are used to touching screens (Ling, 2016).…”

Robust Haebara Linking for Many Groups: Performance in the Case of Uniform DIF

“…For very large sample sizes, more flexible item response functions (e.g., B-spline functions) can be used for linking [61]. Moreover, the estimation of linking constants could be probably made more robust to misspecifications in the IRT model if the first two moments of the trait distribution (i.e., the mean and the standard deviation) instead of item parameters or item response functions are aligned (see [62] for such an approach).…”

Technology and Assessment

“…They inform the graphical design of the entire assessment system, as well as the arrangement of graphical and text elements in individual items. Equivalence studies, such as those undertaken in the development of eTIMSS 2019 are critical (see Fishbein et al 2018). In the year prior to the field trial for eTIMSS 2019, participating countries were invited to administer certain trend items in both paper and electronic format.…”