When Nudges Fail to Scale: Field Experimental Evidence from Goal Setting on Mobile Phones

Abstract

Non-pecuniary incentives motivated by insights from psychology ("nudges") have been shown to be effective tools to change behavior in a variety of fields. An often unanswered question relevant for public policy is whether these promising interventions can be scaled up. In cooperation with a large public utility in Germany, we develop an energy savings application for mobile phones that can be used by the majority of the population. The app randomizes a goal-setting nudge prompting users to set themselves energy consumption targets. The roll-out of the app is promoted by a mass-marketing campaign and large financial incentives. Results document low demand for the energy app in the general population and a tightly estimated null effect of the nudge on electricity consumption among app users. A likely mechanism of the null effect is unfavorable self-selection into the app: users are characterized by an already low baseline energy consumption and exhibit none of the behavioral biases that typically explain why goal setting affects behavior. We also find that the nudge significantly decreases the likelihood to use the app over time. Structural estimates imply that the average user is willing to pay 7.41 EUR to avoid the nudge and the intervention would yield substantial welfare losses if implemented nationwide.

A multi-country meta-analysis on the role of behavioural change in reducing energy consumption and CO2 emissions in residential buildings

“…Our moderator variable analysis points towards possibly lower effects for interventions implemented at scale due to self-selection bias, a concern which has also been noted in primary studies 44 . Our analysis also highlights the need for more long term trials, using rigorous methodology and controls for contiguous factors.…”

Norms, prices, and commitment: A comprehensive overview of field experiments in the energy domain and treatment effect moderators

“…These would presumably depend on whether or not a commitment was made, and, if so, on whether or not the actor successfully completed the behavior or goal to which they committed. The possibility of negative emotions that may be associated especially with public commitment is discussed in Lokhorst et al ( 2015 ), and results in Löschel et al ( 2020 ) illustrate that people may prefer to avoid receiving goal-setting nudges. On the other hand, Baca-Motes et al ( 2013 ) and Joo et al ( 2018 ) found no negative effects of commitment interventions on customer satisfaction in the context of resource conservation campaigns in hotels.…”

“…These can be positive, such as satisfaction, enjoyment and other positive emotional experiences (Herter, 2007 ; Delmas and Lessem, 2014 ; Vesely et al, 2022 ), and increased performance of sustainable behaviors not targeted by the interventions (Steinhorst et al, 2015 ; Carlsson et al, 2021 ; Jessoe et al, 2021 ; see Maki et al, 2019 ; Geiger et al, 2021 for meta-analyses). However, negative effects include avoidance (Löschel et al, 2020 ), motivation crowding out (Lavergne et al, 2010 ; Schwartz et al, 2015 ), adverse behavioral spillover effects (Tiefenbeck et al, 2013 ; McCoy and Lyons, 2017 ; Bjelle et al, 2018 ), negative emotions (Sussman and Gifford, 2012 ) and reactance (Bergquist and Nilsson, 2016 ), as well as economic inefficiencies (Allcott and Greenstone, 2012 ) and problematic distributional effects (Azarova et al, 2019 ; White and Sintov, 2020 ). To find the best approach, interventions thus need to be thoroughly pre-tested prior to their large-scale roll out, and evaluated not only in terms of their intended behavioral impact, but also in terms of their (unintended) downstream effects.…”

On the role of experience in nudge interventions

“…While some recent meta-analyses document a large positive effect of nudge interventions (Beshears & Kosowsky, 2020;DellaVigna & Linos, 2022;Mertens et al, 2022) and partial success (Adams et al, 2022). Other studies document a large body of null effects (e.g., Hummel & Maedche, 2019;Löschel et al, 2020;Narula et al, 2014;Roth, 2020;Steinhart et al, 2013).…”