A pilot design for observational studies: Using abundant data thoughtfully

Abstract

Observational studies often benefit from an abundance of observational units. This can lead to studies that—while challenged by issues of internal validity—have inferences derived from sample sizes substantially larger than randomized controlled trials. But is the information provided by an observational unit best used in the analysis phase? We propose the use of a “pilot design,” in which observations are expended in the design phase of the study, and the posttreatment information from these observations is used to improve study design. In modern observational studies, which are data rich but control poor, pilot designs can be used to gain information about the structure of posttreatment variation. This information can then be used to improve instrumental variable designs, propensity score matching, doubly robust estimation, and other observational study designs. We illustrate one version of a pilot design, which aims to reduce within‐set heterogeneity and improve performance in sensitivity analyses. This version of a pilot design expends observational units during the design phase to fit a prognostic model, avoiding concerns of overfitting. In addition, it enables the construction of “assignment‐control plots,” which visualize the relationship between propensity and prognostic scores. We first show some examples of these plots, then we demonstrate in a simulation setting how this alternative use of the observations can lead to gains in terms of both treatment effect estimation and sensitivity analyses of unobserved confounding.

stratamatch: Prognostic Score Stratification Using a Pilot Design

“…The auto_stratify function in stratamatch divides subjects into strata using a prognostic score (see Hansen (2008)), which summarizes the baseline variation most important to the outcome. In addition to producing strata of more regular size and composition, balancing treatment and control groups based on the prognostic score may confer several statistical benefits: increasing precision (Aikens et al, 2020;Leacy and Stuart, 2014), providing some protection against mis-specification of the propensity score (Leacy and Stuart, 2014;Antonelli et al, 2018), and decreasing the susceptibility of an observed effect to being explained away by unobserved confounding (Rosenbaum and Rubin, 1983;Aikens et al, 2020). However, fitting the prognostic score on the same data set raises concerns of overfitting and may lead to biased effect estimates (Hansen, 2008;Abadie et al, 2018).…”

“…However, fitting the prognostic score on the same data set raises concerns of overfitting and may lead to biased effect estimates (Hansen, 2008;Abadie et al, 2018). For this reason, (Aikens et al, 2020) suggest using a pilot design for estimating the prognostic score.…”

“…The observations in the analysis set can then be stratified based on the quantiles of the estimated prognostic score and matched by propensity score or Mahalanobis distance within strata (see section 2.3). Aikens et al (2020) describe the scenarios in which a prognostic score matching pilot design is most useful. Briefly, the stratamatch approach is best for large data sets (i.e., thousands to millions of observations), especially when the number of control observations is plentiful.…”

“…In the experimental context, these efforts to reduce heterogeneity between compared groups help to increase the precision of the treatment effect estimate. In observational settings, reducing this type of heterogeneity not only improves precision but increases the robustness of the study's conclusions to being explained away by the possibility of unobserved confounding (Rosenbaum, 2005a;Aikens et al, 2020). The stratified matching designin which observations are stratified prior to matching within strata -attempts to emulate the blockrandomized controlled trial design in the observational context in order to secure these statistical benefits over pure propensity score matching.…”

“…This paper discusses the methodological contributions of stratamatch -in particular, the implementation of a novel pilot design approach suggested by Aikens et al (2020) (section 2.2) -and summarizes the package implementation (Section 2.3) with illustrative examples (Section 2.4). While stratamatch may substantially improve the scalability of optimal matching for some large data sets, the main objective of the package is not to implement a computationally complex task but to make sophisticated study design tools and concepts accessible to a wide variety of researchers.…”