Studying Identities with Experiments: Weighing the Risk of Posttreatment Bias Against Priming Effects

Abstract

Scholars from across the social sciences argue that identities – such as race, ethnicity, and gender – are highly influential over individuals’ attitudes, actions, and evaluations. Experiments are becoming particularly integral for allowing identity scholars to explain how these social attachments shape our political behavior. In this letter, we draw attention to how identity scholars should approach the common practice of assessing moderators, measuring control variables, and detecting effect heterogeneity using covariates. Special care must be taken when deciding where to place measures of demographic covariates in identity-related experiments, as these cases pose unique challenges from how scholars traditionally approach experimental design. We argue in this letter that identity scholars, particularly those whose subjects identify as women or minorities, are often right to measure covariates of interest posttreatment.

Increasing Precision in Survey Experiments Without Introducing Bias

“…We also suspect that bias is particularly likely when studies are being conducted on sensitive topics or when respondents may perceive treatment effects as normatively undesirable. For example, researchers are often wary of measuring racial attitudes prior to an experiment out of fear that it may prime racial considerations (Klar, Leeper, and Robison 2019), though some work finds no support for this concern (Valentino, Neuner, and Vandenbroek 2018). In our view, future research would do well to investigate the conditions under which consistency pressures are most likely to influence responding, biasing estimates of treatment effects.…”

“…Additionally, covariates can introduce bias if they are not independent of treatment assignment (Montgomery, Nyhan, and Torres 2018). As a result, some argue that best practice is to rely only on covariates that were measured pretreatment (though see Klar, Leeper, and Robison 2019).…”

The influence of foreign elite rhetoric: National identity, emotions, and attitudes toward austerity

“…; see the Online appendix). This result alleviates concerns over potential post-treatment bias and supports the decision to include these measures post-treatment to avoid possible priming effects, which may occur if identity measures are included pre-treatment (Klar et al., 2020). To account for the multidimensional nature of group identity, I also verified how many participants strongly identify with both, either one, or neither of the two identities (strong identifiers: 75th percentile or above; weak identifiers: 25th percentile or below).…”

Out-of-Control COVID-19 Pandemic Hampers the Nationalism

“…Subjects who completed the survey were compensated $1 and debriefed. All fixed demographic variables except for the partisanship were asked after the treatments (Klar et al, 2020). The treat to the results will be discussed later.…”