Handle with Care: A Sociologist's Guide to Causal Inference with Instrumental Variables

Abstract

Instrumental variables (IV) analysis is a powerful, but fragile, tool for drawing causal inferences from observational data. Sociologists have increasingly turned to this strategy in settings where unmeasured confounding between the treatment and outcome is likely. This paper provides an introduction to the assumptions required for IV and consequences of their violations for applications in sociology. We review three methodological problems IV faces: identification bias (asymptotic bias from assumption violations), estimation bias (finite-sample bias that persists even when assumptions hold), and type-M error (exaggeration of effects given statistical significance). In each case, we emphasize how weak instruments exacerbate these problems and make results sensitive to minor violations of assumptions. Our discussion is informed by a new survey of IV papers published in top sociology journals showing that assumptions often go unstated and robust uncertainty measures are rarely used. We provide a practical checklist to show how IV, despite its fragility, can still be useful when handled with care.

Non-native Accents among School Beginners and Teacher Expectations for Future Student Achievements

“…6 Both instruments are relevant for whether a child speaks with a non-native accent, and they should be uncorrelated with the error terms of the outcome variables. 7 As a result, the instruments should isolate exogenous variation in the accent variable and affect teacher expectations only through their effects on students’ non-native accents (see Felton and Stewart 2022). Unfortunately, the instrumental variable (IV) regressions run into issues of reduced statistical power.…”

The Whiteness of Wealth Management: Colonial Economic Structure, Racism, and the Emergence of Tax Havens in the British Caribbean

“…This identifies a local average treatment effect (LATE) 2 under the assumptions that (a) agricultural suitability causally affects whether countries experienced PIT and/or RG during the early years of tax haven creation (see the data from the main article); (b) there is no common cause of agricultural suitability and PIT/RG status, which is credible because there should be no confounding factors that influence soil quality; (c) the effect of agricultural suitability on tax haven operations today operate only via PIT/RG status-that is, there are no further variables that mediate the effect of agricultural suitability on tax haven operations today; and (d) low agricultural suitability never suppresses (i.e. negatively affects) tax haven operations today (Felton and Stewart 2024). We believe that these are reasonable assumptions.…”

How Much Should We Trust Instrumental Variable Estimates in Political Science? Practical Advice Based on 67 Replicated Studies

“…This is because 2SLS estimates often need to be substantially larger than OLS estimates to achieve statistical significance. This phenomenon is known as Type-M bias and has been discussed in psychology and sociology literature (Felton and Stewart 2022;and Carlin 2014). Invalid instruments exacerbate this issue by providing ample opportunities for generating such large estimates.…”

“…Our work builds on a growing literature evaluating IV strategies in social sciences and offering methods to improve empirical practice. Notable studies include Young (2022), which finds IV estimates to be more sensitive to outliers and conventional t-tests to understate uncertainties; Jiang (2017), which observes larger IV estimates in finance journals and attributed this to exclusion restriction violations and weak instruments; Mellon (2023), which emphasizes the vulnerability of weather instruments; Dieterle and Snell (2016), which develops a quadratic overidentification test and discovers significant nonlinearities in the first stage regression; Felton and Stewart (2022), which finds unstated assumptions and a lack of weak-instrument robust tests in top sociology journals; and Cinelli and Hazlett (2022), which proposes a sensitivity analysis for IV designs in an omitted variable bias framework. This study is the first comprehensive replication effort focusing on IV designs in political science and uses data to shed light on the consequences of a weak first-stage interacting with failures of unconfoundedness or the exclusion restriction.…”