10. Indices of Robustness for Sample Representation

Abstract

Social scientists are rarely able to gather data from the full range of contexts to which they hope to generalize (Shadish, Cook, and Campbell 2002). Here we suggest that debates about the generality of causal inferences in the social sciences can be informed by quantifying the conditions necessary to invalidate an inference. We begin by differentiating the target population into two subpopulations: a potentially observed subpopulation from which all of a sample is drawn and a potentially unobserved subpopulation from which no members of the sample are drawn but which is part of the population to which policymakers seek to generalize. We then quantify the robustness of an inference in terms of the conditions necessary to invalidate an inference if cases from the potentially unobserved subpopulation were included in the sample. We apply the indices to inferences regarding the positive effect of small classes on achievement from the Tennessee class size study and then consider the breadth of external validity. We use the statistical test for whether there is a difference in effects between two subpopulations as a baseline to evaluate robustness, and weThe authors with to acknowledge the helpful comments of two anonymous reviewers, as well as Wei Pan. The opinions expressed in this paper are solely those of the authors. Direct correspondence to Kenneth Frank, 349 350 FRANK AND MIN consider a Bayesian motivation for the indices and compare the use of the indices with other procedures. In the discussion we emphasize the value of quantifying robustness, consider the value of different quantitative thresholds, and conclude by extending a metaphor linking statistical and causal inferences.

Political influences on greenhouse gas emissions from US states

“…Consider a population that is 50% American states and 50% from some other political entity such as European provinces or other administrative divisions, where different causal processes unfold. The correlation between environmentalism and CO 2 emissions would have to be −0.06 in the other entities to invalidate the inference that environmental orientation has an effect on CO 2 emissions in the hypothetical combined data (48). The hypothetical correlation of −0.06 in the other unobserved entities is compared with the estimated correlation of −0.53 between environmental orientation and CO 2 emissions, partialed for covariates, in our data.…”

What Is a “Good” Social Network for Policy Implementation? The Flow of Know-How for Organizational Change

“…There may also be concerns about the external validity of the inferences that potential provider entropy affects changes in implementation because of our relatively small, purposeful, sample. In response, we quantify how much of the estimated effect of potential provider entropy must be due to sampling bias to invalidate our inference (Frank & Min, ). In particular, to invalidate our inference, one would have to replace one‐third (about 7) of our schools with other schools in which there was no effect (Frank et al., ) .…”

Making Sense of Sensitivity: Extending Omitted Variable Bias

“…Although prior work has suggested informal benchmarking procedures using statistics of observed covariates X to help researchers to 'calibrate' their intuitions about the strength of the unobserved confounder Z (Frank, 2000;Imbens, 2003;Frank and Min, 2007;Hosman et al, 2010;Dorie et al, 2016;Carnegie et al, 2016a, b;Middleton et al, 2016;Hong et al, 2018), this practice has undesirable properties and can lead users to erroneous conclusions, even in the ideal case where they do have the correct knowledge about how Z compares with X. This happens because the estimates of how the observed covariates are related to the outcome may be themselves affected by the omission of Z, regardless of whether we assume that Z is independent of X.…”