No abstract

Theory-based evaluations have helped open the 'black box' of programmes. An account is offered of the evolution of this persuasion, through the works of Chen and Rossi, Weiss, and Pawson and Tilley. In the same way as the 'theory of change' approach to evaluation has tackled the complexity of integrated and comprehensive programmes at the community level, it is suggested that a theory-oriented approach based on the practice of realistic cumulation be developed for dealing with the vertical complexity of multi-level governance. K E Y WO R D S : complexity; institutional evaluation hierarchies; integrated programmes; multi-level governance; theory Black Boxes and Evaluation DeficitsSince its inception with the 'War on Poverty' programmes in the US, evaluation has been plagued with the 'black box' problem. The black box is the space between the actual input and the expected output of a programme. Moved by the need to tackle serious social problems, programme designers often gloss over what is expected to happen, the how and why, when an input is put in place; and evaluations do the same concentrating on measuring outputs, whilst attributing the observed difference to the input. All this is hardly informative for a policy design wishing to build upon previous experience. However, for a long time evaluations have coexisted with black box programmes, and have tried to cope with similar shortcomings by developing sophisticated methods for measuring the distance between objectives and results.Nowadays the evaluation community has become more and more concerned with the challenge of how to understand 'what works better for whom in what circumstances, and why ' (Pawson and Tilley, 1997) to improve policy decisions and public sector practice. Theory-oriented approaches reproach the previous, method-oriented approaches for being ineffective, given their inability (or unwillingness) to 'open the black box'. In recent times, theory-oriented approaches have flourished, most notably of the 'theories of change' kind , with respect to the evaluation of complex programmes.

Current debates on impact evaluation have addressed the question 'what works and what doesn't?' mainly focussing on methodology failures when providing evidence of impact. In order to answer that question, this article contrasts different approaches to evaluation in terms of the way they address different kinds of possible failures. First, there is more to be debated than simply methodological failures: there are also programme theory failures and implementations failures. Moreover, not all methodological failures are a simple matter of selection bias. Second, the article reviews issues that have recently been raised within different approaches relative to each failure. For programme theory failure, it is a matter of complexity and providing rival explanations; for implementation failure: how to use guidelines, and how to take context into account; and for methodology failure: how to move from internal to external validity, and to syntheses, within the framework of 'situational responsiveness'. All these issues disclose a terrain for potential exchange between the protagonists of different approaches to impact evaluation.

The US federalist system has been at the origin of evaluation in many ways: providing a laboratory for experimentation of innovative policies, and requiring evaluation of the growing federal intervention from the War on Poverty onwards. Evaluation approaches have been developed that took into consideration the multiple forms of collaboration activated by the federalist system. This experience offers a benchmark for considering evaluation of EU policies and programmes, as it has been elicited by the complex system of EU governance, encompassing both federalist trends -towards concentration of powers at the centre and towards autonomy at the lower levels. First, the article analyses the effects of the centralistic administrative culture of the EU on the way evaluations are conducted at the different levels. Second, it focuses on how the European value added is assessed in the evaluation of two main mechanisms of EU governance that imply a partnership between different jurisdictions: the Structural Funds programmes and the Open Method of Coordination. To do this, it discusses a growing literature of evaluations that utilize theories of power relations, from multilevel governance and network to principal-agent. K E Y W O R D S : additionality; contribution; federalism; subsidiarityAlthough the European project has unfolded as an emergent process in which states once warring among themselves have drawn closer together, its direction is still unclear. It has moved from loose cooperation to stronger integration, from being a small group of six countries to twenty-fi ve partners, along a path toward unity out of diversity. Nevertheless, regardless of how the European political system is defi ned, and of the complexity of its system of governance, it is undeniable, as we shall see, that at the heart of the crisis opened by the 'no' votes of France and the Netherlands to the Constitution, there is a debate about what Van Kersbergen and Veerbeck (2004: 151) call the 'highly controversial f-word of European federalism'.It has now become fashionable to describe the EU political system from the standpoint of its uniqueness, and to avoid any comparison 1 with a federal system, especially the US. Scholars have put forward models of state-centredness Evaluation

No abstract