What Doesn’t Work? Three Failures, Many Answers

Abstract

Current debates on impact evaluation have addressed the question 'what works and what doesn't?' mainly focussing on methodology failures when providing evidence of impact. In order to answer that question, this article contrasts different approaches to evaluation in terms of the way they address different kinds of possible failures. First, there is more to be debated than simply methodological failures: there are also programme theory failures and implementations failures. Moreover, not all methodological failures are a simple matter of selection bias. Second, the article reviews issues that have recently been raised within different approaches relative to each failure. For programme theory failure, it is a matter of complexity and providing rival explanations; for implementation failure: how to use guidelines, and how to take context into account; and for methodology failure: how to move from internal to external validity, and to syntheses, within the framework of 'situational responsiveness'. All these issues disclose a terrain for potential exchange between the protagonists of different approaches to impact evaluation.

No longer lost in translation: the art and science of sports injury prevention implementation research

“…There are many reasons why implemented programmes and their evaluations can fail and these can be summarised broadly as13 14: Programme theory failure: because the intervention is either (A) too complex for its implementation setting or (B) not properly designed to achieve the desired behaviour change. Implementation failure in which the intervention does not adequately address (A) the implementers' own behaviours in relation to intervention delivery or (B) the context in which it is to be delivered. Methodology failure in which (A) internal and/or external validity are compromised, (B) the evaluation plan and tools are not up to the task required of them to demonstrate the outcomes of the intervention or (C) no concurrent process evaluation has been undertaken to explain unexpected observations or to confirm expectations. …”

Effectiveness of a participatory physical and psychosocial intervention to balance the demands and resources of industrial workers: A cluster-randomized controlled trial

“…The failure of PIPPI to improve recovery and work ability could either be explained by a theory failure (ie, the program theory being erroneous) or implementation failure (ie, inadequately designed or implemented), or a combination of both (52). However, it seems unlikely that the lack of effects can be attributed to implementation failure alone due to the fact that (i) 100% of the planned intervention workshops were delivered, (ii) there was a moderate-to-high participation rate (61-75%), (iii) workers identified many issues to be addressed and planned several action plans to solve these issues, and (iv) a high percentage of intervention group workers reported being satisfied with the activities (83%) and that the action plans for solving the issues were implemented as planned (79%).…”

Realistic evaluation as an avenue to learning for development NGOs

“…These differences have implications for the perceived role of the goals, mechanisms and contexts in different methodologies. In the goal-oriented approach (Scriven, 1981;Stame, 2010) prerequisites include the goals being clear, implementation following precise protocols and evaluation being able to reveal impact. In contrast, goal-free approaches subscribe to 'bounded rationality' according to which programmes work through mechanisms that are enacted by situated actors by exploiting favourable opportunities; hence the results are linked to the process (Stame, 2010: 373).…”