Ecological validity is vital to experimental research because designs that are too artificial may not speak to any real-world political phenomenon. One such concern is treatment self-selection: if individuals in the real world self-select treatments, such as political communications, how well does the sample average treatment effect estimate the effects of message exposure for those individuals who would -if given the choice -opt-in to and out of receiving treatment? This study shows that randomization masks effect heterogeneity between individuals who would select different messages if given the choice. Yet such selections are themselves complex, revealing additional challenges for realistically studying treatments prone to self-selection. The evidence of effect heterogeneity raises questions about the appropriateness of random assignment experiments for studying political communication and the results more broadly advance our understanding of citizens' selection into and responses to communications when, as they often do, have choice over what messages to receive. * An earlier version of this paper was presented at the Midwest Political Science Association 2012 Annual Meeting, Chicago, IL and the International Society for Political Psychology 2012 Annual Meeting, Chicago, IL. Thanks are due to John Brehm, Jamie Druckman, Samara Klar, Jeff Mondak, Marco Steenbergen, and the editorial team at JEPS for feedback on various stages of the project, and to Jamie, Fay Lomax Cook, and Toby Bolsen for sharing survey data from the t1 panel wave.
1Experiments are seen as a gold standard because randomized assignment offers unparalleled internal validity. An important and less studied concern is ecological validity, or the "realism" of the experiment. Ecological validity is vital because studies that are too artificial may not speak to any real world political phenomenon. In particular, the recognition that in the real world individuals self-select their own "treatments" raises a concern that randomized exposure may not constitute a realistic exploration of political communication (
Effects of Self-Selected Political CommunicationsArguments conveyed by media and political elites are widely seen as an important source of information that citizens can use when forming preferences (Disch 2011;Chong and Druckman 2007). Evidence of this influence comes largely from experiments that expose participants to different messages and measure effects on argument evaluations, opinions, issue importance, information-seeking, etc. (see Ansolabehere et al. 1994;Arceneaux and Johnson 2012;Berinsky and Kinder 2006;Brewer and Gross 2005;Iyengar and Kinder 1987;Miller and Krosnick 2000;Nelson, Clawson, and Oxley 1997;Petty and Cacioppo 1986). Questions have been raised, however, about these kind of studies given that, in Hovland's words: "In an experiment the audience on whom the effects are being evaluated is one which is fully 2 exposed to the communication. On the other hand, in naturalistic situations with which surveys are typically conce...