How to make causal inferences using texts

Abstract

Text as data techniques offer a great promise: the ability to inductively discover measures that are useful for testing social science theories with large collections of text. Nearly all text-based causal inferences depend on a latent representation of the text, but we show that estimating this latent representation from the data creates underacknowledged risks: we may introduce an identification problem or overfit. To address these risks, we introduce a split-sample workflow for making rigorous causal inferences with discovered measures as treatments or outcomes. We then apply it to estimate causal effects from an experiment on immigration attitudes and a study on bureaucratic responsiveness.

Causal inference from text: A commentary

“…How does showing someone an ad affect how they write about a product? Analyzing text data to answer these types of causal questions is the problem addressed in the excellent article of Egami et al ( 5 ).…”

“…What should we look for in a textual code that is used for causal inference? Inspired by the framework developed by Egami et al ( 5 ), we look further at the different relationships that a code can have with the text. These ways of thinking could lead to further research and methods for causal inference from text.…”

“…In the example from Egami et al ( 5 ), the data are from an open-ended survey about immigration. The treatment involves information about an immigrant’s criminal history; the response is a textual description of what the respondent thinks should happen to the immigrant.…”

“…The work by Egami et al ( 5 ) advances the field of text analysis and causality by showing how to estimate causal quantities from text data, where the text is either an outcome or a treatment. The methods developed in their paper—and in the growing literature on causal inference from text—can help social scientists and other researchers make new types of causal inferences from text data.…”

Seeded Topic Models in Digital Archives: Analyzing the Swedish Understanding of Immigration, 1945--2019

“…We demonstrate that widely recognized, emotion-laden events have immediate but no lasting effects on interpretations of immigration and that it takes prolonged periods of rupture to change them. This event analysis exemplifies how researchers can use the output of computational text models to trace causal claims within, for example, the traditional regression model framework (Egami et al 2018, Roberts et al 2020, Fong & Grimmer 2019, Feder et al 2021, Gencoglu & Gruber 2020.…”