Systematic mapping of climate and environmental framing experiments and re-analysis with computational methods points to omitted interaction bias

Abstract

Ambitious climate policy requires acceptance by millions of people whose daily lives would be affected in costly ways. In turn, this requires an understanding of how to get the mass public on board and prevent a political backlash against costly climate policies. Many scholars regard ‘framing’, specially tailored messages emphasizing specific subsets of political arguments to certain population subgroups, as an effective communication strategy for changing climate beliefs, attitudes, and behaviors. In contrast, other scholars argue that people hold relatively stable opinions and doubt that framing can alter public opinion on salient issues like climate change. We contribute to this debate in two ways: First, we conduct a systematic mapping of 121 experimental studies on climate and environmental policy framing, published in 46 peer-reviewed journals and present results of a survey with authors of these studies. Second, we illustrate the use of novel computational methods to check for the robustness of subgroup effects and identify omitted interaction bias. We find that most experiments report significant main and subgroup effects but rarely use advanced methods to account for potential omitted interaction bias. Moreover, only a few studies make their data publicly available to easily replicate them. Our survey of framing researchers suggests that when scholars successfully publish non-significant effects, these were typically bundled together with other, significant effects to increase publication chances. Finally, using a Bayesian computational sparse regression technique, we offer an illustrative re-analysis of 10 studies focusing on subgroup framing differences by partisanship (a key driver of climate change attitudes) and show that these effects are often not robust when accounting for omitted interaction bias.