Multisite Causal Mediation Analysis in the Presence of Complex Sample and Survey Designs and Non-Random Non-Response

Abstract

Summary This study provides a template for multisite causal mediation analysis using a comprehensive weighting‐based analytic procedure that enhances external and internal validity. The template incorporates a sample weight to adjust for complex sample and survey designs, adopts an inverse probability of treatment weight to adjust for differential treatment assignment probabilities, employs an estimated non‐response weight to account for non‐random non‐response and utilizes a propensity‐score‐based weighting strategy to decompose flexibly not only the population average but also the between‐site heterogeneity of the total programme impact. Because the identification assumptions are not always warranted, a weighting‐based balance checking procedure assesses the remaining overt bias, whereas a weighting‐based sensitivity analysis further evaluates the potential bias related to omitted confounding or to propensity score model misspecification. We derive the asymptotic variance of the estimators for the causal effects that account for the sampling uncertainty in the estimated weights. The method is applied to a reanalysis of the data from the National Job Corps Study.

Unpacking Complex Mediation Mechanisms and Their Heterogeneity Between Sites in a Job Corps Evaluation

“…These methodological advances are important for the current study: We expect that the impacts of education and training on earnings may differ between the Job Corps setting and a control setting if participants receive education and training of a higher quality in Job Corps than in other alternative programs. We also expect, as indicated by past research (Qin et al., 2019), that these impacts may differ between individuals and between sites. To reduce reliance on model specifications, Lange, Rasmussen, and Thygesen (2014) extended RMPW to the case of multiple concurrent mediators.…”

“…To incorporate the sampling variability of the estimated weights, we extend an estimation procedure proposed by Qin et al. (2019) based on the GMM framework. While the two estimation steps are presented sequentially for expository purposes, in practice, we estimate them jointly through GMM.…”

“…These questions have not been addressed empirically in past evaluations of Job Corps. Researchers have typically combined education and training into a single measure (e.g., Flores & Flores‐Lagunes, 2013; Qin & Hong, 2017; Qin et al., 2019). Hence it has been unclear whether Job Corps’ investment in GED is worthwhile especially in light of the recent debate concerning the economic value of obtaining a GED certificate (Heckman & LaFontaine, 2006; Tyler, 2003).…”

“…However, to our knowledge, there have been no formal scholarly discussions about the unique research opportunities and methodological challenges that arise in investigations of complex mediation mechanisms that may vary across sites. Extending the RMPW approach, recent work by Qin and Hong (2017) and by others (Bein et al., 2018; Hong, Deutsch, & Hill, 2015; Hong, Qin, & Yang, 2018; Qin et al., 2019) has advanced methods for causal mediation analysis accompanied by sensitivity analysis in single‐level and multi‐level settings. Yet these new advances have focused on a single mediator.…”

Did you Conduct a Sensitivity Analysis? A New Weighting-Based Approach for Evaluations of the Average Treatment Effect for the Treated

“…Although this paper primarily considers a binary treatment and a continuous outcome, the same rationale can be extended to studies of a treated group with multiple untreated comparison groups and to categorical outcomes. A similar logic has been applied to weighting‐based SA in causal mediation analysis (Hong, Qin, & Yang, 2018; Qin et al, 2019).…”