Measuring Widening Proficiency Differences in International Assessments: Are Current Approaches Enough?

Abstract

Participation in international large‐scale assessments has grown over time with the largest, the Programme for International Student Assessment (PISA), including more than 70 education systems that are economically and educationally diverse. To help accommodate for large achievement differences among participants, in 2009 PISA offered low‐performing systems the option of including an easier set of items in the assessment with an aim of providing improved achievement estimates. However, there remains a lack of evidence on the performance of this design innovation. As such, we simulate a design that closely mirrors the PISA 2015 math assessment in order to empirically examine the benefits of including easy items for low‐performing countries. We extend the PISA design to include increased numbers of easy items and items that are easier than currently implemented. Findings show that the current PISA approach provides little advantage compared to a common test for all participants. Our study also demonstrates persistent bias, low coverage rates, and low correlations between generating and estimated proficiency under current designs. Through our simulation we also show that to improve achievement estimation for low performers about half of the items would need to be made significantly easier.

Routing Strategies and Optimizing Design for Multistage Testing in International Large‐Scale Assessments

“…This move is particularly important in low‐performing countries where average student proficiency is well below the international average. Without some type of adaptive testing the vast majority of students in these systems receive questions that are too difficult, offering little opportunity to engage with items and potentially resulting in biased achievement estimates for the lowest performers (Rutkowski, Rutkowski, & Liaw, ). In response, PISA is implementing a multistage test design for the 2018 cycle.…”

Analysing PIAAC Data with Structural Equation Modelling in Mplus

“…With the increasing number of countries, language groups, and educational systems participating in international large-scale assessments comes the challenge of establishing that the measures used for comparisons are sufficiently invariant (Rutkowski et al 2018). However, the commonly used approach of multi-group CFA to establish measurement invariance across many groups may increase the chances of falsely detecting non-invariance due to the large number of pairwise comparisons of model parameters (Rutkowski and Svetina 2013).…”

Commentary on Special Issue Articles

“…The authors explain why MST is to be preferred over itemlevel adaptation and go on to explain how implementing MST results in improved accuracy especially at the tails of the proficiency distribution. This is important not only because of within-country variation, but also because of the variation among participating countries-exactly the issue that concerns D. Rutkowski et al (2018)! A noteworthy distinction between PISA and PIAAC is that in the former no prior information is available to guide the selection of the first item set, while in the latter such information is collected at the outset and, therefore, can be utilized to advantage.…”

“…The article by D. Rutkowski, Rutkowski, and Liaw () directly confronts the question of how to maintain accuracy of measurement in the face of increasing heterogeneity. In particular, these authors examine the efficacy of the strategy adopted by PISA to develop item clusters comprising easier items and to rotate these clusters through the booklet design.…”

“…Based on PISA 2015 results, D. Rutkowski et al. () selected 30 countries for their simulation: 10 lower proficiency, 10 medium proficiency, and 10 high proficiency. Using PISA data, as well as information from the PISA technical report, they constructed a simulation database that emulates PISA results for mathematics.…”