Generalizability of heterogeneous treatment effect estimates across samples

Coppock, Alexander; Leeper, Thomas J.; Mullinix, Kevin J.

doi:10.1073/pnas.1808083115

Cited by 335 publications

(190 citation statements)

References 43 publications

Supporting

Mentioning

170

Contrasting

Unclassified

Order By: Relevance

“…MTurkers also tend to skew liberal, and so our sample may have underestimated initial polarization. Generally, however, analyses of political research find that research on nonrepresentative samples such as MTurk typically replicates well on nationally representative samples (30), suggesting our experimental results are likely to replicate. A second concern about generalizability is ecological validity, i.e., whether our experiment reflects the dynamics of political belief formation more broadly.…”

Section: Discussionmentioning

confidence: 69%

The wisdom of partisan crowds

Becker

Porter

Centola

2019

Proc. Natl. Acad. Sci. U.S.A.

130

View full text Add to dashboard Cite

Theories in favor of deliberative democracy are based on the premise that social information processing can improve group beliefs. While research on the "wisdom of crowds" has found that information exchange can increase belief accuracy on noncontroversial factual matters, theories of political polarization imply that groups will become more extreme-and less accurate-when beliefs are motivated by partisan political bias. A primary concern is that partisan biases are associated not only with more extreme beliefs, but also with a diminished response to social information. While bipartisan networks containing both Democrats and Republicans are expected to promote accurate belief formation, politically homogeneous networks are expected to amplify partisan bias and reduce belief accuracy. To test whether the wisdom of crowds is robust to partisan bias, we conducted two web-based experiments in which individuals answered factual questions known to elicit partisan bias before and after observing the estimates of peers in a politically homogeneous social network. In contrast to polarization theories, we found that social information exchange in homogeneous networks not only increased accuracy but also reduced polarization. Our results help generalize collective intelligence research to political domains. collective intelligence | polarization | networks | the wisdom of crowds | deliberative democracy

show abstract

Section: Discussionmentioning

confidence: 69%

The wisdom of partisan crowds

Becker

Porter

Centola

2019

Proc. Natl. Acad. Sci. U.S.A.

130

View full text Add to dashboard Cite

show abstract

“…We utilize the Amazon Mechanical Turk (MTurk) platform in order to achieve the desired statistical power across all variations of the experiment. Recent literature points to the robustness, generalizability, and reproducibility of laboratory findings in online environments (Arechar et al, 2018;Coppock et al, 2018;Snowberg and Yariv, 2018). 2 Results from our high-powered study suggest a resounding null effect with tight confidence intervals.…”

Section: Introductionmentioning

confidence: 63%

Requiem for a Nudge: Framing effects in nudging honesty

Dimant

Kleef

Shalvi

2020

Journal of Economic Behavior & Organization

View full text Add to dashboard Cite

We examine framing effects in nudging honesty, in the spirit of the growing norm-nudge literature, by utilizing a high-powered and pre-registered study. Across four treatments, participants received one random truthful norm-nudge that emphasized 'moral suasion' based on either what other participants previously did (empirical message) or approved of doing (normative message) and varied in the framing (positive or negative) in which it was presented. Subsequently, participants repeatedly played the 'mind game' in which they were first asked to think of a number, then rolled a digital die, and then reported whether the two numbers coincide, in which case a bonus was paid. Hence, whether or not the report was truthful remained unobservable to the experimenters. We find compelling null effects with tight confidence intervals showing that none of the norm-nudge interventions worked. A follow-up experiment reveals the reason for these convincing null-effects: the information norm-nudges did not actually change norms. Notably, our secondary results suggest that a substantial portion of individuals misremembered norm-nudges such that they conveniently supported deviant behavior. This subset of participants indeed displayed significantly higher deviance levels, a behavior pattern in line with literature on motivated misremembering and belief distortion. We discuss the importance of this high-powered null finding for the flourishing norm-nudge literature and derive policy implications. JEL-Codes: B410, D010, D900.

show abstract

“…Because the use of convenience samples, such as MTurk, can raise questions about generalizability, recent research has investigated the correspondence between experimental results obtained from MTurk and other samples (see Berinsky, Huber, and Lenz ). Research comparing treatment effects from experiments conducted with population‐based samples to those from experiments conducted in MTurk generally shows high levels of correspondence (see Coppock ; Coppock, Leeper, and Mullinix ; Jeong et al ; Mullinix et al ). There is also evidence that liberals and conservatives on MTurk share the views of their ideological counterparts in the U.S. population (Clifford, Jewell, and Waggoner ).…”

Section: Survey Experimentsmentioning

confidence: 99%

Does Administrative Burden Influence Public Support for Government Programs? Evidence from a Survey Experiment

Keiser

Miller

2019

Public Administration Review

View full text Add to dashboard Cite

Research indicates that administrative burden influences the behaviors and views of clients and potential clients of government programs. However, administrative burden may also shape mass attitudes toward government programs. Taking a behavioral public administration approach, the authors consider whether and how exposure to information about administrative burden embedded within eligibility-based programs influences citizen favorability toward those programs. It is hypothesized that if information about the existing screening mechanisms is highlighted and made salient, this will lead to greater approval of eligibility-based programs. This expectation is evaluated using a survey experiment that explores administrative burden in the Temporary Assistance for Needy Families (TANF) program. The evidence shows that being exposed to information about administrative burden increases favorability toward TANF and its recipients, though these effects are conditional on party identification. The results provide insight into a potential consequence of administrative burden, showing the way in which information regarding burden can shape citizens' support for eligibility-based programs. Evidence for Practice• Public managers in social welfare programs face challenges in gaining public support because of the stigma associated with these programs. • The evidence suggests that giving the public information about program screening improves views toward welfare programs. • Increasing awareness about program screening processes may be beneficial. However, public officials should consider potential trade-offs, such as discouraging applications. Susan M. Miller is associate professor of political science at the University of South Carolina. Broadly, her research explores the consequences of institutional and program design for policy implementation and outcomes.

show abstract

Generalizability of heterogeneous treatment effect estimates across samples

Cited by 335 publications

References 43 publications

The wisdom of partisan crowds

The wisdom of partisan crowds

Requiem for a Nudge: Framing effects in nudging honesty

Does Administrative Burden Influence Public Support for Government Programs? Evidence from a Survey Experiment

Contact Info

Product

Resources

About