Experimentation in policy design: insights from the building sector

Abstract

The current article questions how experimentation in policy-design plays out in practice. In particular, it is interested in understanding how the content and process of policy-design experiments affect their outcomes. The article does so by building on an original study into 31 real-world examples of experimentation in policy-design in the building sector in Australia, the Netherlands and the United States. All examples aim to improve the environmental sustainability of the building sector. The article finds that these 31 examples have attracted moderate to substantial numbers of participants (policy outcome HO.i), but have not achieved substantial numbers of buildings built or retrofitted with high levels of sustainability (policy outcome HO.ii). By carefully unpacking these policy-designs into a number of key characteristics it finds that this mismatch between the two outcomes may partly be explained by flawed policy-design processes. The article concludes with the main lessons learnt, and provides some suggestions on how to improve experimentation in policy-design.

Exploring governance learning: How policymakers draw on evidence, experience and intuition in designing participatory flood risk planning

“…The research contributes to wider discussions on participatory and collaborative environmental governance, evidence-based policy and governance, (adaptive) policy learning and policy transfer. We seek to advance the debate in that we deliberately depart from the traditional focus of the policy learning (and related) literature on the content of policy to focus on procedural dimensions and the process of planning and governance ( Emerson and Gerlak, 2014 , van der Heijden, 2013 ).…”

Unpack the black box of pilot sampling in policy experimentation: A qualitative comparative analysis of China's public hospital reform

“…Experiments in the policy world hardly follow the rules for laboratory settings (Ko & Shin, 2017; van der Heijden, 2014). Governments do enjoy a great deal of discretion in setting the terms and conditions of piloting design (Jowell, 2003; Nair, 2020).…”

“…“ (P)olicy experimentation is different from controlled or lab‐based experiments because often the political process of government involved in these experiments cannot be equated to the usual components of an experiment, such as having a hypothesis that can be tested through repeated trials, control groups, and randomization ” (Nair, 2020, p. 347). Political constraints, ethical considerations, or both often make it extremely hard to carry out randomized controlled trials in the world of public policy (Jowell, 2003; van der Heijden, 2014). Often used interchangeably with “policy experiment”, policy pilot is not defined narrowly as a methodological approach based on a control‐treatment group or randomization design––instead, it emphasizes the use of pilots as a tool for gaining relevant evidence and useful knowledge for policymaking in the face of vast uncertainties (Ko & Shin, 2017; Nair & Howlett, 2016).…”

“…This is not a trivial matter of concern, because “sampling strategy” is of strategic importance in an experiment and fundamentally affects both internal validity and external validity, in common scientific language. We do know that randomization and controllability in lab settings are neither realistic nor suitable for real‐world public policies (Ko & Shin, 2017; Nair, 2020; van der Heijden, 2014), but we know little regarding the criteria and considerations that national governments use when they sample subnational “subjects” in an experimental program. Do they strive to secure “representativeness” by recruiting a sample that is as diverse as possible?…”

Experimental governance for low-carbon buildings and cities: Value and limits of local action networks

“…Finally, these scholars argue for a wider repertoire of governance instruments than traditional government-led direct interventions-such as regulation, subsidies, and taxes. By including market based approaches and incentives-such as benchmarking, information sharing, media attention-highly localised governance instruments can be developed that are of specific interest to local actors (Evans, 2011;Van der Heijden, 2014a). At city level governance, experimentation is considered as particularly promising because of scaling possibilities:…”