Inference without randomization or ignorability: A stability‐controlled quasi‐experiment on the prevention of tuberculosis

Abstract

The stability‐controlled quasi‐experiment (SCQE) is an approach to study the effects of nonrandomized, newly adopted treatments. While covariate adjustment techniques rely on a “no unobserved confounding” assumption, SCQE imposes an assumption on the change in the average nontreatment outcome between successive cohorts (the “baseline trend”). We provide inferential tools for SCQE and its first application, examining whether isoniazid preventive therapy (IPT) reduced tuberculosis (TB) incidence among 26 715 HIV patients in Tanzania. After IPT became available, 16% of untreated patients developed TB within a year, compared with only 0.5% of patients under treatment. Thus, a simple difference in means suggests a 15.5 percentage point (pp) lower risk (p ≪ .001). Adjusting for covariates using numerous techniques leaves this effectively unchanged. Yet, due to confounding biases, such estimates can be misleading regardless of their statistical strength. By contrast, SCQE reveals valid causal effect estimates for any chosen assumption on the baseline trend. For example, assuming a baseline trend near 0 (no change in TB incidence over time, absent this treatment) implies a small and insignificant effect. To argue IPT was beneficial requires arguing that the nontreatment incidence would have risen by at least 0.7 pp per year, which is plausible but far from certain. SCQE may produce narrow estimates when the plausible range of baseline trends can be sufficiently constrained, while in every case it tells us what baseline trends must be believed in order to sustain a given conclusion, protecting against inferences that rely upon infeasible assumptions.

Credible learning of hydroxychloroquine and dexamethasone effects on COVID-19 mortality outside of randomized trials

“…The key mathematical fact is that in this case, for any assumed baseline trend, we can estimate the treatment effect experienced by the treated patients , without additional assumptions or covariates [1, 2]. For intuition behind this result, we return to the natural experiment considered above, where no individual in the first cohort takes the treatment, and hence the average outcome we observe is the “average non-treatment outcome”.…”

“…Finally, rather than place our confidence in a single assumption, we “invert” the analysis to reveal the needed assumptions about the baseline trend in mortality to declare that a given treatment had a beneficial, null, or harmful effect . Confidence intervals can be constructed for the effect estimate at any given choice of the baseline trend assumption, using the approach described in [2]. Throughout this paper, we describe an estimated effect as a “significantly” beneficial or harmful effect when its 95% confidence interval excludes zero, which is equivalent to a two-sided p-value at or below 0.05.…”

“…This study employs the stability-controlled quasi-experiment (SCQE) [1,2] approach to investigate treatment effects on COVID-19 patients, which differs from conventional approaches for observational studies in two key ways. First, unlike standard covariate-adjustment strategies (regression, matching, weighting, and stratification), SCQE does not rely on the assumption that there are no unobserved confounders, i.e.…”

“…This does change the interpretation of effect in terms of the population of patients to whom it applies; see[2] 3. Analyses using only aggregate data of this kind can be conducted using web-based software available at https: //amiwulf.shinyapps.io/SCQE_demo/.…”

“… 2 This does change the interpretation of effect in terms of the population of patients to whom it applies; see [2]. …”