Analyzing data of a multilab replication project with individual participant data meta-analysis: A tutorial

Abstract

Multi-lab replication projects such as Registered Replication Reports and Many Labs projects are used to replicate an effect in different labs. Data of these projects are usually analyzed using conventional meta-analysis methods. This is a valid approach, but it does not make optimal use of the available data as summary data are analyzed rather than the participant data. I propose to analyze data of multi-lab replication projects by using individual participant data (IPD) meta-analysis where the participant data are analyzed directly. Two important advantages of IPD meta-analysis are that it generally has larger statistical power to detect moderator effects and allows drawing conclusions at the participant rather than lab level. I illustrate IPD meta-analysis using the published Registered Replication Report by McCarthy et al. (2018) and provide R code and offer recommendations to facilitate researchers to apply these methods to their own data.

Toward Cumulative Cognitive Science: A Comparison of Meta-Analysis, Mega-Analysis, and Hybrid Approaches

“…This is a point that has been extensively documented in previous methodological literature (e.g., Burke et al, 2017 ; Legha et al, 2018 ; Riley et al, 2020 ), which contrasts typical meta-analyses, which involve two stages (first estimation of effects within studies, then across studies), against IPD meta-analyses, called in this context one-stage models (direct integration of data from several studies in a single model). This point has also been highlighted in the psychological literature, in the context of multi-lab large-scale replications (van Aert, 2020 —who also provides a tutorial for conducting IPD meta-analyses). Previous methodological and simulation work has demonstrated that meta-analyses and IPD meta-analyses converge when shared assumptions are met (Burke et al, 2017 ; Papadimitropoulou et al, 2019 ).…”

Analyzing Data of a Multilab Replication Project With Individual Participant Data Meta-Analysis

“…The parameter τ 2 reflects the between-study variance in true effect size and indicates whether the lab's true effect sizes θ i are all the same (homogeneous) or different from each other (heterogeneous). Heterogeneity in true effect size can be explained by extending the statistical model in (1) to a random-effects meta-regression model where study characteristics are included as moderators (e.g., Thompson & Sharp, 1999;Van Houwelingen et al, 2002). That is, a lab's true effect size becomes a regression equation in a random-effects meta-regression model (e.g., β 0 + β 1 x where x is a moderator variable).…”

“…The study was replicated in 22 labs and the total sample size was 7,373 (see McCarthy et al, 2018 for more details). All analyses were conducted in the statistical software R (Version 4.1.0, R Core Team, 2021), the R package papaja (Aust & Barth, 2020) was used for writing the article, and annotated R code to analyze the RRR is available in the supplemental materials at the Open Science Framework (OSF; Van Aert, 2019a: https://osf.io/c9zep/).…”

“…Before fitting the random-effects model to the RRR, I first computed the raw mean differences and corresponding sampling variances for each lab (see Van Aert, 2019a: https://osf.io/c9zep/). I used the R package (Version 3.0.2, Viechtbauer, 2010) for fitting the random-effects model.…”

A Primer on Synthesizing Individual Participant Data Obtained From Complex Sampling Surveys: A Two-Stage IPD Meta-Analysis Approach

“…For instance, accessing raw participant data rather than summary data allows for testing relevant research questions while increasing statistical power to detect individual-level interactions and modeling the sources of within-and between-study variance. Furthermore, IPD meta-analysis represents a tool for evaluating replication in multi-lab studies (van Aert, 2020).…”