How Does Treatment Self-Selection Affect Inferences About Political Communication?

Abstract

Ecological validity is vital to experimental research because designs that are too artificial may not speak to any real-world political phenomenon. One such concern is treatment self-selection: if individuals in the real world self-select treatments, such as political communications, how well does the sample average treatment effect estimate the effects of message exposure for those individuals who would -if given the choice -opt-in to and out of receiving treatment? This study shows that randomization masks effect heterogeneity between individuals who would select different messages if given the choice. Yet such selections are themselves complex, revealing additional challenges for realistically studying treatments prone to self-selection. The evidence of effect heterogeneity raises questions about the appropriateness of random assignment experiments for studying political communication and the results more broadly advance our understanding of citizens' selection into and responses to communications when, as they often do, have choice over what messages to receive. * An earlier version of this paper was presented at the Midwest Political Science Association 2012 Annual Meeting, Chicago, IL and the International Society for Political Psychology 2012 Annual Meeting, Chicago, IL. Thanks are due to John Brehm, Jamie Druckman, Samara Klar, Jeff Mondak, Marco Steenbergen, and the editorial team at JEPS for feedback on various stages of the project, and to Jamie, Fay Lomax Cook, and Toby Bolsen for sharing survey data from the t1 panel wave. 1Experiments are seen as a gold standard because randomized assignment offers unparalleled internal validity. An important and less studied concern is ecological validity, or the "realism" of the experiment. Ecological validity is vital because studies that are too artificial may not speak to any real world political phenomenon. In particular, the recognition that in the real world individuals self-select their own "treatments" raises a concern that randomized exposure may not constitute a realistic exploration of political communication ( Effects of Self-Selected Political CommunicationsArguments conveyed by media and political elites are widely seen as an important source of information that citizens can use when forming preferences (Disch 2011;Chong and Druckman 2007). Evidence of this influence comes largely from experiments that expose participants to different messages and measure effects on argument evaluations, opinions, issue importance, information-seeking, etc. (see Ansolabehere et al. 1994;Arceneaux and Johnson 2012;Berinsky and Kinder 2006;Brewer and Gross 2005;Iyengar and Kinder 1987;Miller and Krosnick 2000;Nelson, Clawson, and Oxley 1997;Petty and Cacioppo 1986). Questions have been raised, however, about these kind of studies given that, in Hovland's words: "In an experiment the audience on whom the effects are being evaluated is one which is fully 2 exposed to the communication. On the other hand, in naturalistic situations with which surveys are typically conce...

Making Austerity Popular: The Media and Mass Attitudes toward Fiscal Policy

“…Importantly, we estimate what would happen if everyone were exposed to a particular message. We do not estimate the impact of the message on those who would choose to be exposed to it through their selection of paper (Leeper 2017).…”

Political Knowledge and Misinformation in the Era of Social Media: Evidence From the 2015 UK Election

“…The fact that we find an effect of this size is notable given that (as shown below) the impact of tweets on knowledge varies according to the source of the tweets, thus making this a conservative estimate of the potential effect of exposure to topic-specific information on Twitter to improve knowledge. 11 Leeper (2017) demonstrates the limitations of a randomized trial in estimating this quantity of interest. Calculating the average treatment effect of media exposure over an entire sample can mask significantly heterogeneous treatment effects.…”

Persuading the Enemy: Estimating the Persuasive Effects of Partisan Media with the Preference-Incorporating Choice and Assignment Design

“…Partisan news can be persuasive to these individuals. Separately estimating these effects for people with different media-viewing preferences can help detect important heterogeneity in the effects of partisan media (e.g., Gaines and Kuklinski 2011; Leeper 2017). We demonstrate that reading partisan news on a politically salient topic can change people’s opinions, as well as their actions, to different degrees depending on their media preferences.…”

“…The PICA design builds upon the general framework known in the medical literature as the patient preference trial (PPT). As noted, some existing political science research employs PPT frameworks, incorporating participants' preferences over treatment options (Torgerson and Sibbald, 1998) to examine the effects of experimental treatments that may vary depending on those preferences (Gaines and Kuklinski, 2011;Leeper, 2017). This step makes it possible to avoid relying on self-reported ideology to identify people's media preferences.…”