Experimentalism and development evaluation: Will the bubble burst?

Abstract

Bridging the current divide of opinion about experimentalism would help protect an evaluation brand currently under threat in international evaluation circles. In order to help settle a lingering and unnecessary controversy, this opinion article describes the policy force field that triggered the recent surge of interest in experimental methods in development evaluation; digs up the historical and philosophical roots of the long-standing epistemological debate; outlines the value and boundaries of experimentalism; and speculates about its prospects in development evaluation.

Assessing the Effectiveness of Projects Supporting On-Farm Conservation of Native Crops: Evidence From the High Andes of South America

“…It could be argued that to generate the rigorous evidence required to justify the implementation OFC projects, there may be a need to use randomized control trials (RCTs). RCTs are increasingly used and advocated for to test agricultural interventions (e.g., Duflo, Kremer, & Robinson, 2008, 2011Farley, Lucas, Molyneaux, Penn, & Hogue, 2012), but there is also recognition of their limitations (Barahona, 2010;Barrett & Carter, 2010;Picciotto, 2012). To our knowledge RCTs have not been applied to OFC projects and given the complexity of the interventions, the contextual and heterogeneous nature of this type of project, the reliance of project implementation on the interest, motivation and capacity to choose of participants, factors that limit the relevance and feasibility of RCTs (Barrett & Carter, 2010), suggest that their application to this type of project would be challenging.…”

Using case studies to explore the external validity of ‘complex’ development interventions

“…2 See, among others, Cartwright (2007), Deaton (2010), Picciotto (2012), Ravallion (2009) and Shaffer (2011). Nobel laureate James Heckman has been making related critiques of 'randomization bias' in the evaluation of social policy experiments for over twenty years.…”

Some Thoughts on Development Evaluation Processes

“…Stern et al (2012) for different perspectives on this discussion and with a particular focus on impact evaluations. 2 See for example Ling (2012) and Picciotto (2012) for introductions to recent discussions, including to the distinction between simple, complicated and complex interventions. 3 Many stakeholders, including staff from donor evaluation units, seem to agree -at least informally -that there is a problem with the quality of many development evaluations (see for example Picciotto 2011: 171 and 2012: 220, which explicitly mention the low quality of many development evaluations.…”