The contribution of process tracing to theory-based evaluations of complex aid instruments

Abstract

This article focuses on methodological challenges in evaluating complex program aid interventions like budget support. We show that recent innovations in process-tracing methodology can help solve the identified challenges and increase the strength of causal inference made when using case studies in demanding settings. For the specific task of evaluating the governance effectiveness of budget support interventions, we developed a more fine-grained causal mechanism for a subset of the comprehensive program theory of budget support. Moreover, based on the informal use of Bayesian logic, we have elaborated on how to increase the conclusiveness of empirical evidence for one part of the theorized causal mechanism. We argue that by establishing an explicit theorized mechanism prior to empirical research and by critically judging our evidence according to an informal Bayesian logic we can remedy some of the problems at hand in much case-study research and increase the inferential leverage in complex within-case evaluation studies.

Theory of Change Analysis: Building Robust Theories of Change

“…Theories of change are the basis for theory-based evaluation approaches, such as logical analysis (Brousselle & Champagne, 2011;Rey et al, 2012), realist evaluation (Blamey & Mackenzie, 2007;Pawson, 2013), contribution analysis (Mayne, 2012), and process tracing (Schmitt & Beach, 2015). As such, the robustness of the theory of change used matters.…”

How We Model Matters: A Manifesto for the Next Generation of Program Theorizing

“…Yet a third strategy is to pursue analytical approaches such as process trac ing, whereby more fine-grained causal mechanisms can be examined and verifi ed ( Schmitt & Beach, 2015 ). A key strength of process tracing is that it relies on clear ly specified tests, whereby the presence or absence of hypothesized mechanisms can be determined.…”

“…emphasis on reflective practice emerges in part from a broader push in evaluation circles toward evaluative thinking ( Vo & Archibald, 2018 ). More specifi cally, the notion of reflective practice is rooted in the recent work by Th omas Schwandt (2015 ), who describes the reflective practitioner as someone who develops the knowledge and skills to design and implement methodologies in a variety of un familiar situations, has the capacity to comprehend local contexts, knows how to adapt methodological principles and practices accordingly, and has the ability to determine courses of action and move forward in a way that balances methodo logical, practical, and ethical considerations.…”

Integrating theory-based evaluation and process tracing in the evaluation of civil society gender budget initiatives

“…TBE approaches, however, do not make explicit the causal links between events in the causal chain from the intervention's inputs to the intended outcome, leaving us in the dark about the actual causal forces responsible for producing the outcome (Befani and Mayne, 2014;Delahais and Toulemonde, 2012;Schmitt and Beach, 2015). A review of evaluations conducted to assess the relevance and effectiveness of UN Women's work in GRB between 2008 and 2011 reveals the use of a descriptive TBE approach, with the focus placed on evaluating stated and implicit assumptions that affect programme development (UN Women, 2010).…”

“…The three approaches, however, do not explicitly unpack the 'causal links in causal processes': they either treat underlying mechanisms as unobservable (RE); or as 'assumptions instead of as vital parts of the causal mechanism that should be studied empirically' (CA) (Schmitt and Beach, 2015: 430-431); or are silent about causal mechanisms (GEM). PTM (discussed in the next section) is increasingly gaining popularity as an alternative approach that provides a solution to this problem, thereby enabling stronger causal inferences to be made about complex interventions (Befani and Mayne, 2014;Schmitt and Beach, 2015;Stern et al, 2012). PTM, like CA, acknowledges that a particular outcome may be produced by different mechanisms.…”