No abstract

Purpose, significance of research and theoretical frame work In the last decades we encountered rapid developments in information and communication technologies. The inclusion of the worldwide web into daily life brought new and important implications also for education. Most of the schools and educational systems started providing extensive computer networks for their students and these are increasingly becoming main components of the teaching and learning environment, but so far little is known about the effectiveness and use of these technologies (Fraillon et al. 2014). Conclusions from research carried out in the field are partly contradictory. Many authors who examined computer use and student achievement found they were positively related

Background Standard methods for analysing data from large-scale assessments (LSA) cannot merely be adopted if hierarchical (or multilevel) regression modelling should be applied. Currently various approaches exist; they all follow generally a design-based model of estimation using the pseudo maximum likelihood method and adjusted weights for the corresponding hierarchies. Specifically, several different approaches to using and scaling sampling weights in hierarchical models are promoted, yet no study has compared them to provide evidence of which method performs best and therefore should be preferred. Furthermore, different software programs implement different estimation algorithms, leading to different results. Objective and method In this study, we determine based on a simulation, the estimation procedure showing the smallest distortion to the actual population features. We consider different estimation, optimization and acceleration methods, and different approaches on using sampling weights. Three scenarios have been simulated using the statistical program R. The analyses have been performed with two software packages for hierarchical modelling of LSA data, namely Mplus and SAS. Results and conclusions The simulation results revealed three weighting approaches performing best in retrieving the true population parameters. One of them implies using only level two weights (here: final school weights) and is because of its simple implementation the most favourable one. This finding should provide a clear recommendation to researchers for using weights in multilevel modelling (MLM) when analysing LSA data, or data with a similar structure. Further, we found only little differences in the performance and default settings of the software programs used, with the software package Mplus providing slightly more precise estimates. Different algorithm starting settings or different accelerating methods for optimization could cause these distinctions. However, it should be emphasized that with the recommended weighting approach, both software packages perform equally well. Finally, two scaling techniques for student weights have been investigated. They provide both nearly identical results. We use data from the Programme for International Student Assessment (PISA) 2015 to illustrate the practical importance and relevance of weighting in analysing large-scale assessment data with hierarchical models.

The International Association for the Evaluation of Educational Achievement (IEA) undertakes international large-scale assessments (ILSAs) to provide reliable measures of student achievement in the context of learning. These ILSAs can be used to assess the quality and equity of education systems and enable countries to make informed decisions for improving education. Surveying all individuals belonging to the target population (e.g., students, classes, or teachers) would be a costly enterprise. The ILSA design instead focuses on measuring high-quality representative samples, ensuring the results are close to the true values in the populations and can be usefully compared across cultures and countries. Sampling is thus a key element in establishing the validity and reliability of the results obtained from cross-national large-scale assessments. This chapter reviews IEA's sampling strategies, from defining the target populations, to compiling sampling frames, applying complex sampling methods, and accounting for the methodology when analyzing data. These concepts are explained for a non-technical audience in an illustrative way. Common methodologies used when selecting random samples for IEA studies include stratification, multiple-stage and cluster sampling, and unequal selection probabilities. The samples yielded by implementing any one of these methods call for specific methods of data analysis that take account of the sampling design. Sampling weights have to be applied to arrive at accurate conclusions on population features when using sample data; specific methods are needed to compute standard errors and confidence intervals correctly. This comprehensive guide to IEA's methodology illustrates the vital importance of sampling in providing high-quality, reliable data that are generalizable to the targeted populations in an unbiased manner.

No abstract