Assessing Threats to Inference with Simultaneous Sensitivity Analysis: The Case of US Supreme Court Oral Arguments

Abstract

Political scientists relying on observational data face substantial challenges in drawing causal inferences. A particularly problematic threat to inference is the unobserved confounder. As a means to assess this threat, we introduce simultaneous sensitivity analysis to the political science literature. As an application, we consider the potentially confounded relationship between Supreme Court justice voting and oral argument quality. We demonstrate that this relationship is sensitive to the presence of a confounder, to a degree that threatens inference, and explore the confounder both theoretically and empirically. More generally, we show how sensitivity analysis can guide inquiry related to a covariate that cannot be directly measured.

Justice-level heterogeneity in certiorari voting: US Supreme Court October terms 1939, 1968, and 1982

“…A strong argument can also be made for US Petitioner as a legal covariate. The most direct evidence is from Johnson et al (2006: 107), which shows that legal arguments from the Solicitor General and from attorneys representing the federal government are higher quality than those from other lawyers; for evidence that this holds not only at the merits stage, but also on cert, see Budziak and Lempert (2018: 47). The categorization of amicus briefs is more contested.…”

Quantitative bias analysis in practice: review of software for regression with unmeasured confounding

“…In the past decade, there have been several reviews of QBA methods [ 2 , 5 , 9 – 18 ]. Of these, three papers reviewed software implementations of QBA methods: [ 18 ] gave an overview of QBA to unmeasured confounding and a tutorial on the newly released R package tipr [ 19 ], the supplementary of [ 13 ] provided a brief summary of software implementing Rosenbaum-style QBA methods [ 20 ], and [ 11 ] reviewed software implementations before its publication in July 2014. Also, comparisons of QBA methods have primarily been limited to analyses with a binary outcome [ 10 , 21 – 28 ].…”

Quantitative bias analysis in practice: Review of software for regression with unmeasured confounding

“…Lack of knowledge about QBA, and of analyst-friendly methods and software have been identified as barriers to the widespread implementation of a QBA [7][8][9] . In the past decade, there have been several reviews of QBA methods 2,5,[9][10][11][12][13][14][15][16][17] . Only two of these papers reviewed software implementations of QBA methods: the supplementary of 15 provided a brief summary of software implementing Rosenbaum-style QBA methods 18 , and 11 reviewed software implementations before its publication in July 2014.…”