Towards Appropriate Impact Evaluation Methods

Abstract

The choice of evaluation methods is one of the most plagued questions for evaluators (Szanyi, Azzam, & Galen, 2012). Especially in development evaluation, where interventions tend to be very complex, and multiple stakeholders hold competing interests (Holvoet et al., 2018), this question is pressing. While one can discern an emerging consensus among evaluation scholars that not only (quasi-) experimental evidence can lay monopoly claim to the production of the best effectiveness evidence (Stern et al., 2012), this idea is definitely not yet commonly shared among all evaluators, let alone among commissioners of impact evaluation studies. The article by Wendy Olsen presents a strong persuasive case for considering alternative impact evaluation methods that can help overcoming the shortcomings of Randomized Controlled Trials (RCTs). The question is, however, on which conditions one should opt for such alternative methods. Or to put it differently: on which conditions can it be "unwise" to resort to such methods in impact evaluations? The aim of this contribution is to bring some nuance into the methods debate, by drawing attention to the broader organizational and institutional context in which impact evaluations take place. The commentary revolves around the idea that the choice of a particular evaluation method will as much be affected by considerations of technical appropriateness as well as political appropriateness. With technical appropriateness, we refer to the ability of the chosen method to answer the impact evaluation question at stake. Political appropriateness in turn concerns the fit between the broader institutional setting and specific impact evaluation methods. Any assessment of the merit of particular evaluation methods should ideally consider both angles.

Policy Effectiveness through Configurational and Mechanistic Lenses: Lessons for Concept Development

“…Experimental designs such as Randomized Controlled Trials (RCTs), which rely on counterfactual comparisons between situations with ("policy on") and without the intervention ("policy off"), are particularly suited for this objective. While experimental evidence can be very insightful for policy makers, particularly for accountability purposes (Pattyn 2019), not all policy settings lend themselves to the application of RCTs. Importantly, the "policy works" claim relies on the assumption that the intervention is the primary cause of the effect of interest (Stern et al 2012, p. 38).…”

Warum initiieren Minister Evaluationen? Eine Fallstudie der flämischen Regierung in Belgien

“…For the administration in charge of the implementation of the evaluations announced in the policy notes and the evaluation community at large, our findings can be read as an incentive to engage in policy evaluations that are not primarily accountability focused, but that also enable policy learning. In fact, not all evaluation methods lend themselves to policy learning (Pattyn 2019). This is not to say, on the other hand, that parliamentarians cannot use learning-oriented evaluations to hold ministers accountable (Speer et al 2015;Bundi 2016).…”

The use of Qualitative Comparative Analysis (QCA) to address causality in complex systems: a systematic review of research on public health interventions

“…There is no perfect evaluation design for achieving these aims. As in other fields, the choice of design will in part depend on the availability of counterfactuals, the extent to which the investigator can control the intervention, and the range of potential cases and contexts [75], as well as political considerations, such as the credibility of the approach with key stakeholders [76]. There are inevitably 'horses for courses' [77].…”