Sampling weights in multilevel modelling: an investigation using PISA sampling structures

Abstract

Background Standard methods for analysing data from large-scale assessments (LSA) cannot merely be adopted if hierarchical (or multilevel) regression modelling should be applied. Currently various approaches exist; they all follow generally a design-based model of estimation using the pseudo maximum likelihood method and adjusted weights for the corresponding hierarchies. Specifically, several different approaches to using and scaling sampling weights in hierarchical models are promoted, yet no study has compared them to provide evidence of which method performs best and therefore should be preferred. Furthermore, different software programs implement different estimation algorithms, leading to different results. Objective and method In this study, we determine based on a simulation, the estimation procedure showing the smallest distortion to the actual population features. We consider different estimation, optimization and acceleration methods, and different approaches on using sampling weights. Three scenarios have been simulated using the statistical program R. The analyses have been performed with two software packages for hierarchical modelling of LSA data, namely Mplus and SAS. Results and conclusions The simulation results revealed three weighting approaches performing best in retrieving the true population parameters. One of them implies using only level two weights (here: final school weights) and is because of its simple implementation the most favourable one. This finding should provide a clear recommendation to researchers for using weights in multilevel modelling (MLM) when analysing LSA data, or data with a similar structure. Further, we found only little differences in the performance and default settings of the software programs used, with the software package Mplus providing slightly more precise estimates. Different algorithm starting settings or different accelerating methods for optimization could cause these distinctions. However, it should be emphasized that with the recommended weighting approach, both software packages perform equally well. Finally, two scaling techniques for student weights have been investigated. They provide both nearly identical results. We use data from the Programme for International Student Assessment (PISA) 2015 to illustrate the practical importance and relevance of weighting in analysing large-scale assessment data with hierarchical models.

Linguistic diversity and students’ reading achievements in developing countries: a hierarchical linear model analysis using PISA-D data

“…PISA-D data is similar to most international educational assessments, as cognitive outcome variables are provided as plausible values (PV). We estimated models for each of the ten PVs provided in the data while including school weight, and the average over these ten calculations provided more robust estimates (Mang et al 2021, Rubin 1984. We checked model fit by using the deviance statistic (-2 X log likelihood), where a larger deviance for a more complex model indicates a poor fit of that model relative to the less complex one (Anderson 2012).…”

School leadership and achievement gaps based on socioeconomic status: a search for socially just instructional leadership

“…Scholars also recommend the use of sample weights due to the unequal probability of selection (Lorah, 2018), but it is largely unclear which weights should be used and how to achieve robust results (Laukaityte and Wiberg, 2018). As it has been shown for PISA 2015, “using only the school weights provide the most unbiased estimates for hierarchical models” (Mang et al. , 2021, p. 36), we employed school weights (W_SCHGRNRABWT) to correct for imperfections due to the difference between the sample and the reference population.…”

Are instructional practices different between East and West? An analysis of Grade 8 TIMSS 2019 data

“…The author(s) received no financial support for the research, authorship, and/or publication of this article. Mang et al (2021). 4.…”