A Primer on Synthesizing Individual Participant Data Obtained From Complex Sampling Surveys: A Two-Stage IPD Meta-Analysis Approach

Abstract

The increasing availability of individual participant data (IPD) in the social sciences offers new possibilities to synthesize research evidence across primary studies. Two-stage IPD meta-analysis represents a framework that can utilize these possibilities. While most of the methodological research on two-stage IPD meta-analysis focused on its performance compared with other approaches, dealing with the complexities of the primary data has received little attention, particularly when IPD are drawn from complex sampling surveys. Complex sampling surveys often feature clustering, stratification, disproportionate sampling, and multiple stages of sample selection to obtain nationally or internationally representative data from a target population. Furthermore, IPD from these studies are likely to provide more than one effect size. To address the complexities of the primary and meta-analytic data obtained from complex surveys, we propose a two-stage IPD meta-analytic approach and illustrate its utility. To aid meta-analysts who wish to utilize complex survey data, we present a sequence of steps and discuss the methodological decisions and options within. Given its flexibility and ability to deal with the complex nature of the primary data, the proposed two-stage approach opens up new analytical possibilities for synthesizing knowledge meta-analytically.

Digital Gender Gaps in Students’ Knowledge, Attitudes and Skills: An Integrative Data Analysis Across 32 Countries

“…We adopted a two-stage individual participant data (IPD) meta-analysis approach to analysing and synthesizing the student data from ICILS 2013 and 2018. This two-stage IPD meta-analytic approach generates the effect sizes of interest from each study independently using the same analysis protocol and then combines the resultant effect size estimates using meta-analytic models (Campos et al, 2021;Scherer et al, 2021). Please find the respective code and output in the Open Science Framework project at https://osf.io/6um4t/?view_only=1ff8ba652cc646d1b90930b2a8fbee57.…”

“…A more balanced and comprehensive coverage of content domains could shed further light on the specific areas in which digital gender divides occur. This could be achieved by extending our integrative data analysis to a meta-analysis that combines ICILS data with aggregated data from published primary studies (Campos et al, 2021).…”

Role of ILSAs for Meta-Analyses in Education

“…Given that the raw primary data are oftentimes not available, meta-analysts may have to trust the estimation and reporting of the effect sizes in the publication and have hardly any chance to perform further adjustments. However, such adjustments are possible for most ILSA data-in fact, if the raw data of primary studies are available, the meta-analysts are in full control of the effect size estimation and can estimate them and the respective sampling (co-)variances from analytic models that incorporate the complex survey design features of ILSAs, such as multilevel models with sampling weights, stratifying variables, plausible values, and multi-group structures (Campos et al, 2021). Overall, meta-analysts have at least two options to address the complex survey design, especially the nested data structure, in primary studies: (a) Adjust the reported effect sizes by the 𝐼𝐶𝐶 !…”

“…Such situations offer the possibility to generate effect sizes and sampling (co-)variances from the same kind of analytic model. Recently, some ways to meta-analyze only ILSA data have been proposed (e.g., Brunner et al, 2022;Campos et al, 2021) with respective examples (Blömeke et al, 2021;Keller et al, 2022).…”