Theory-Based Evaluation and Types of Complexity

Abstract

Theory-based evaluations have helped open the 'black box' of programmes. An account is offered of the evolution of this persuasion, through the works of Chen and Rossi, Weiss, and Pawson and Tilley. In the same way as the 'theory of change' approach to evaluation has tackled the complexity of integrated and comprehensive programmes at the community level, it is suggested that a theory-oriented approach based on the practice of realistic cumulation be developed for dealing with the vertical complexity of multi-level governance. K E Y WO R D S : complexity; institutional evaluation hierarchies; integrated programmes; multi-level governance; theory Black Boxes and Evaluation DeficitsSince its inception with the 'War on Poverty' programmes in the US, evaluation has been plagued with the 'black box' problem. The black box is the space between the actual input and the expected output of a programme. Moved by the need to tackle serious social problems, programme designers often gloss over what is expected to happen, the how and why, when an input is put in place; and evaluations do the same concentrating on measuring outputs, whilst attributing the observed difference to the input. All this is hardly informative for a policy design wishing to build upon previous experience. However, for a long time evaluations have coexisted with black box programmes, and have tried to cope with similar shortcomings by developing sophisticated methods for measuring the distance between objectives and results.Nowadays the evaluation community has become more and more concerned with the challenge of how to understand 'what works better for whom in what circumstances, and why ' (Pawson and Tilley, 1997) to improve policy decisions and public sector practice. Theory-oriented approaches reproach the previous, method-oriented approaches for being ineffective, given their inability (or unwillingness) to 'open the black box'. In recent times, theory-oriented approaches have flourished, most notably of the 'theories of change' kind , with respect to the evaluation of complex programmes.

Key strategies to improve systems for managing patient complaints within health facilities – what can we learn from the existing literature?

“…First, limited understanding of contexts of effective interventions . There is a growing recognition of the importance of the context in ensuring the success of complex health systems interventions [41,117–122], including improvements to patient complaints systems. While specific components of the interventions are usually well described, it is often unclear which contextual facilitators and barriers affected their effectiveness, and how.…”

Conceptual and Methodological Issues in Evaluations of Road Safety Countermeasures

“…The problem arises, however, when no significant results are found, despite the use of rigorous design and methodology. Blackbox or methods-driven research that can make it a challenge for policy makers to understand how and why an intervention does or does not work and to identify the key elements for successful implementation [7,16,26,30,59].…”

“…That said, most evaluations of road safety countermeasures do not use theories by which to conceptualize and design the intervention and its evaluation, and to interpret results [14,24]. From a conceptual standpoint, programs, interventions, treatments and policies are based on the validity of underlying assumptions about how and why they are supposed to work [7][8][9]12,13,[25][26][27][28][29][30]. However, as Elvik [13] writes: "Most road safety evaluation studies evaluate the effects of a certain road safety measure on the number of accidents, accident rate or the severity of injuries without describing how and why a certain safety measure produces the observed changes in accident occurrence or injury severity" (p. 741).…”

“…The use of theory in evaluation to identify causal mechanisms or chains (also called processes, links, assumptions, pathways or models), through which an intervention is expected to achieve better outcomes and the use of process evaluation has been widely adopted for health, human, and community services, and a range of international development programs, such as the World Bank for evaluating humanitarian services [17,19,21,[25][26][27][28][29][30][35][36][37][38][39]. For example, the International Initiative for Impact Evaluation (3ie) guide states: "Studies should clearly lay out how it is that the intervention (inputs) is expected to affect final outcomes, and test each link (assumption) from inputs to outcomes (sometimes referred to as the program theory).…”

A theory-based evaluation of a community-based funding scheme in a disadvantaged suburban city area

“…Theory-based approaches have become an increasingly popular means of examining how and why a given community-based initiative works (Stame, 2004). The advantages of this approach are that it allows evaluators to elucidate, in detail, program processes and causal factors for observed effects, particularly when outputs and outcomes are not easily identifiable or directly measurable.…”