How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling

Abstract

Policymakers are increasingly turning to insights gained from the experimental method as a means to inform large scale public policies. Critics view this increased usage as premature, pointing to the fact that many experimentally-tested programs fail to deliver their promise at scale. Under this view, the experimental approach drives too much public policy. Yet, if policymakers could be more confident that the original research findings would be delivered at scale, even the staunchest critics would carve out a larger role for experiments to inform policy. Leveraging the economic framework of Al-Ubaydli et al. (2019), we put forward 12 simple proposals, spanning researchers, policymakers, funders, and stakeholders, which together tackle the most vexing scalability threats. The framework highlights that only after we deepen our understanding of the scale up problem will we be on solid ground to argue that scientific experiments should hold a more prominent place in the policymaker's quiver.

What matters for the scalability of prejudice reduction programs and interventions?

“…This was in line with the assessment factors of Zamboni et al (11). Context for successful scaling of prejudice reduction interventions goes beyond the delivery-related factors that Al-Ubaydli et al (5) proposed. While they alluded, in passing, that representativeness of the situation may include issues such as political opposition, the examples that they focused on were largely related to delivery -whether the delivery, dosage and program were correct.…”

“…The recent work of Al-Ubaydli et al (6) is an excellent example of a holistic approach to scaling social science research in the real world. The model Al-Ubaydli and colleagues proposed pointed to three main barriers to scalability in small-scale, successful interventions: inference, representativeness of the population, and representativeness of the situation (5,6). First, the authors suggest that there may be issues with the reliability of the evidence, due to the lack of replication of the effects, insufficient power in the studies, and incorrect interpretation of the p-value of results.…”

Cash incentives for weight loss work only for males

“…Recently, behavioral scientists and economists have raised their attention to the role of demographics for their critical relevance to the success of scaled implementation of behavioral interventions. Specifically, the assumption that an intervention would work equally well for all demographic groups is a major cause of 'voltage drop'the phenomenon that the effect size of an intervention drops significantly relative to that reported in the original research when it is implemented at a large scale in the society (Banerjee et al, 2018;Al-Ubaydli et al, 2019a, 2019bHo et al, 2021). The same could occur for large-scale weight loss programs that use financial incentives to motivate weight loss, because financial incentives for weight loss work only for one gender.…”

“…An important consideration concerning such uses of financial incentives is whether the incentives work equally for everyone, specifically between genders. Having an empirical answer to this question is important: It will push policy makers to reconsider the provision of incentives for weight loss as an effective policy for all (see Al-Ubaydli et al, 2019a, 2019bHo et al, 2021 for a discussion of the voltage drop problem) and motivate behavioral scientists to design gender-specific interventions for solving the obesity problem. In this report, we provide evidence that financial incentives work differently for the two genders.…”

How Experiments with Children Inform Economics

“…examine the market for knowledge and show that the incentives of the various actors (researchers, government policymakers, citizens) themselves can lead to the scale-up problem. Importantly, if researchers backward induct when setting up their original research plans to ensure accurate and swift transference of programs to scale, not only are initial insights useful, but they also aid in the generality (external validity) problem Al-Ubaydli et al (2021). provide a checklist for both researchers and policymakers that involves design choice from idea inception to roll out at scale.31…”