Design Experiments: Engaging Policy Makers in the Search for Evidence about What Works

Abstract

This article presents an argument for the greater use of design experiments, which can assist policy making because they provide both robust and timely evidence. We discuss their origins in education research, set out the methodology and propose some adaptations to the techniques used in these education studies to foster their application to a range of policy fields and problem areas. Design experiments need to meet two challenges. Can they provide valid evidence? Can they provide evidence that will be used by policy makers? Our argument shows how design experiments are robust when set against the classical canons of scientific study. We further claim that the design experiment approach offers a more viable means to developing evidence‐based policy making than other forms of evaluation because of the timeliness of the insights that it provides.

The Dilemmas of Citizen Inclusion in Urban Planning and Governance to Enable a 1.5 °C Climate Change Scenario

“…Low-income communities in Indore, India, were also able to use experiments to test implementation pathways, prioritise climate actions, and evaluate overall project benefits (Chu, 2016b). Although some have challenged their external validity and replicability, experiments have been shown to be a good approach for encouraging intensive dialogue and smallscale innovation (Stoker & John, 2009).…”

“…In-creasingly applied to low-carbon transition policies, for example, experiments promote overall decision-making effectiveness and help to generate new governance capacities. Methodologically, experiments can support evidence-based policy-making by supplying robust evaluations and opportunities to redesign existing approaches (Stoker & John, 2009). Experiments can therefore be seen as 'laboratories' of learning and sharing best practices (Karvonen & van Heur, 2014), which allow diverse actors to embed emerging needs and priorities into urban plans (Evans, 2011).…”

The Possible Experts: How Epistemic Communities Negotiate Barriers to Knowledge Use in Ecosystems Services Policy

“…In short, experts need to ensure their work is 'useable' (Lindblom and Cohen, 1979). This means using accessible communication strategies that fit the context (Howlett, 2009;Jewell and Bero, 2008;Yetley, 2007 (Stoker, 2010;Stoker and John, 2009). Although such enthusiasm may be tempered by evidence that decision-makers -even North America where the experimental method is most common -are not always aware of RCT data (Bedard and Ouimet, 2012).…”

The Simple, the Complicated, and the Complex: Educational Reform Through the Lens of Complexity Theory

“…Where a pilot programme must often succeed the first time or be abandoned, a policy experiment requires patience to allow time for lessons to be learned from each iteration and for redesigns to be implemented. It does not require a reform to be exactly right the first time but accepts the need for adjustment and improvement as the initiative develops (Stoker and John, 2009). It will be a gradual, communal process that requires strong leaders and broad consensus based on commitment to work, talk, analyse and revisit the way to success (Eppel et al, 2011).…”

“…The barriers between design and implementation should be collapsed to create a single iterative process, driven by feedback to core decision makers from the local level actors, with the goal creating cycles of interaction between decision makers and those charged with implementation, who are most often the actors responsible for turning general ideas into practical strategies. All stakeholders, at all levels, should listen to and interact with each other to continuously refine and improve the reform (Stoker and John, 2009;James, 2006). The traditional model of policy making, in which design precedes and is distinct from implementation, should be replaced by one of experimentation, an approach explored more thoroughly later in this paper, in which end-of-programme summative assessments are supplemented by continuous formative assessments (Magro and Wilson, 2013) in what can be characterised as "an oscillating multi-level interaction" (Heilmann, 2008, p. 12).…”