Optimal Matching for Observational Studies That Integrate Quantitative and Qualitative Research

Abstract

A quantitative study of treatment effects may form many matched pairs of a treated subject and an untreated control who look similar in terms of covariates measured prior to treatment. When treatments are not randomly assigned, one inevitable concern is that individuals who look similar in measured covariates may be dissimilar in unmeasured covariates. Another concern is that quantitative measures may be misinterpreted by investigators in the absence of context that is not recorded in quantitative data. When text information is automatically coded to form quantitative measures, examination of the narrative context can reveal the limitations of initial coding efforts. An existing proposal entails a narrative description of a subset of matched pairs, hoping in a subset of pairs to observe quite a bit more of what was not quantitatively measured or automatically encoded. A subset of pairs cannot rule out subtle biases that materially affect analyses of many pairs, but perhaps a subset of pairs can inform discussion of such biases, perhaps leading to a reinterpretation of quantitative data, or perhaps raising new considerations and perspectives. The large literature on qualitative research contends that open-ended, narrative descriptions of a subset of people A c c e p t e d M a n u s c r i p tcan be informative. Here, we discuss and apply a form of optimal matching that supports such an integrated, quantitative-plus-qualitative study. The optimal match provides many closely matched pairs plus a subset of exceptionally close pairs suitable for narrative interpretation. We illustrate the matching technique using data from a recent study of police responses to domestic violence in Philadelphia, where the police report includes both quantitative and narrative information.

From Feature Importance to Distance Metric: An Almost Exact Matching Approach for Causal Inference

“…Further, the units within a matched group can be far from each other in covariate space -i.e., the matched groups are generally not auditable [Parikh et al, 2022b]. To date, the only observational causal inference techniques that attempt to optimize accuracy while maintaining auditability are those stemming from the almost-matchingexactly (AME) framework, namely optimal matching (optMatch) [Yu et al, 2021, Kallus, 2017, genetic matching (GenMatch) [Diamond and Sekhon, 2013], FLAME/DAME [Wang et al, 2017, Dieng et al, 2019, MALTS [Parikh et al, 2022b[Parikh et al, , 2019[Parikh et al, , 2022a and AHB [Morucci et al, 2020] algorithms. FLAME/DAME can scale to extremely large datasets but handles only categorical variables.…”

“…Auditability allows domain experts to validate the estimation procedure, argue about the violation of key assumptions, and determine whether the analysis is trustworthy. Parikh et al [2022a] and Yu et al [2021] showed that the audit of matched groups using external unstructured data is crucial in high-stakes healthcare and social science scenarios. Since causal analyses often depend on untestable assumptions, it is critical to determine whether all important confounders are accounted for, if data are processed correctly, and whether the treatment and control units in the matched groups are cohesive enough to be comparable [Parikh et al, 2022c].…”