The Effect of Sample Weighting on Hierarchical Linear Modeling in the Large-Scale Assessment Data

Abstract

This study examines how the different uses of sampling weights in the analysis of TIMSS 2019 data affects the proportion of variance in student achievement explained by schools and the estimation of standard errors. The research sample comprises 227,345 8th-grade students from 7,636 schools in 39 countries. Mathematics achievement and science achievement are considered separately as output variables in all 39 countries. All plausible values are included in the analysis. Four weighting scenarios are examined: no weighting, weighting at only level 1, weighting at only level 2, and weighting at both levels. In total, 312 models are established and examined. According to the research results, the coefficients, standard errors, reliabilities, and Chi-square estimations change depending on how the weighting variable is handled in the models, and as a result, the portion of variance in the output variable arising from the differences between schools also changes. The portion attributable to between-school differences can reach up to 20% in some countries. Therefore, researchers modeling hierarchical data using HLM are advised to plan how they handle the weighting variable prior to conducting the study.