Meta-Analyzing Individual Participant Data From Studies With Complex Survey Designs: A Tutorial on Using the Two-Stage Approach for Data From Educational Large-Scale Assessments

Abstract

Descriptive analyses of educational phenomena are a vital component of educational research. Such analyses yield reliable results when using representative individual participant data (IPD) from educational large-scale assessments (ELSAs). The meta-analytic integration of these results offers unique and novel research opportunities to provide strong empirical evidence of the consistency and generalizability of educational phenomena. This paper offers methodological guidance on how to use the two-stage approach to IPD meta-analysis to account for the statistical complexities of ELSAs (e.g., sampling weights, clustered and missing IPD, dependent effect sizes), first, for conducting descriptive analyses (Stage 1), and second, for integrating the results with meta-analytic and meta-regression models (Stage 2). The two-stage approach is illustrated with IPD on reading achievement from PISA in order to integrate standardized mean differences (e.g., gender differences), correlations (e.g., with students’ SES), and interactions between educational characteristics at the participant level (e.g., the interaction between gender and SES).

Differentiation hypotheses of intelligence: A systematic review of the empirical evidence and an agenda for future research.

“…In the second step, the results from applying these methods can be meta-analyzed (e.g., Tucker-Drob & Bates, 2016), for example, by focusing on the parameters depicting differentiation effects to learn about their size and direction. Such an individual participant data meta-analysis (Brunner et al, 2021; McArdle & Horn, 1985, 2002; Riley & Fisher, 2021) would enable rapid acquisition of a robust, quantitatively integrated empirical base for differentiation effects. Of course, conducting an individual participant meta-analysis involves a great deal of effort in obtaining and analyzing the individual data sets, but it would advance the research field considerably (see also Graham et al, 2022).…”

PROTOCOL: School‐based language, math, and reading interventions for executive functions in children and adolescents: A systematic review

“…As training EF has limited or no effects on academic skills, how is the robust associations between measures of EF and academic achievement to be explained? One hypothesis is that academic skills and EF are both caused by one or more variables, which are associated with both academic skills (Brunner, 2021; Sirin, 2005) and EF (Jacob, 2015; Lawson, 2018). One example of such a variable may be parental socioeconomic status (SES).…”

Role of ILSAs for Meta-Analyses in Education

“…For instance, when meta-analysts are interested in deriving the correct effect size measures for gender differences in achievement, the standardized mean difference (𝑆𝑀𝐷) may be the effect size of their choice (Borenstein et al, 2009). When computing 𝑆𝑀𝐷, the pooled standard deviation can incorporate information about the nesting of the primary study data (e.g., students nested in classrooms or schools; Brunner et al, 2022). Hedges (2007) proposed several ways to incorporate the intraclass correlation 𝐼𝐶𝐶 !…”

“…Such situations offer the possibility to generate effect sizes and sampling (co-)variances from the same kind of analytic model. Recently, some ways to meta-analyze only ILSA data have been proposed (e.g., Brunner et al, 2022;Campos et al, 2021) with respective examples (Blömeke et al, 2021;Keller et al, 2022).…”