Combining Theory-Driven Evaluation and Causal Loop Diagramming for Opening the ‘Black Box’ of an Intervention in the Health Sector: A Case of Performance-Based Financing in Western Uganda

Abstract

Increased attention on “complexity” in health systems evaluation has resulted in many different methodological responses. Theory-driven evaluations and systems thinking are two such responses that aim for better understanding of the mechanisms underlying given outcomes. Here, we studied the implementation of a performance-based financing intervention by the Belgian Technical Cooperation in Western Uganda to illustrate a methodological strategy of combining these two approaches. We utilized a systems dynamics tool called causal loop diagramming (CLD) to generate hypotheses feeding into a theory-driven evaluation. Semi-structured interviews were conducted with 30 health workers from two districts (Kasese and Kyenjojo) and with 16 key informants. After CLD, we identified three relevant hypotheses: “success to the successful”, “growth and underinvestment”, and “supervision conundrum”. The first hypothesis leads to increasing improvements in performance, as better performance leads to more incentives, which in turn leads to better performance. The latter two hypotheses point to potential bottlenecks. Thus, the proposed methodological strategy was a useful tool for identifying hypotheses that can inform a theory-driven evaluation. The hypotheses are represented in a comprehensible way while highlighting the underlying assumptions, and are more easily falsifiable than hypotheses identified without using CLD.

Effects in spite of tough constraints - A theory of change based investigation of contextual and implementation factors affecting the results of a performance based financing scheme extended to malnutrition in Burundi

“…As in several recent contributions [46][47][48][49], our study adopted a broad view on the ToC of PBF. Our case shows that the intervention itself (PBF-N) interplayed with at least three other pre-existing interventions, each having its own ToC: delivery of nutrition services at the HC level, supervision by the district team and PBF-FHC.…”

Is quality affordable for community health systems? Costs of integrating quality improvement into close-to-community health programmes in five low-income and middle-income countries

“…The health impacts of integrating QI are harder to attribute due to the complex, iterative and locally driven nature of the approach. Measuring and attributing the downstream benefits of a service delivery intervention that are intrinsically valuable to a decision maker or population is challenging 20 68–72. Adding to the challenges of potential confounding, in ‘Step Six’ of the intervention (figure 1), QI teams have the freedom to design and test QI interventions to address locally relevant problems they select (in contrast to having a standard QI intervention imposed by higher-level or external stakeholders).…”

Effect of results-based financing on facility-based maternal mortality at birth: an interrupted time-series analysis with independent controls in Malawi

“…Last, we need to acknowledge the limited generalisability of our findings to other RBF settings. Unlike most other RBF programmes where payments linked to quantity aspects of service delivery dominate,34 the RBF4MNH kept a stronger focus on payments linked to quality of care processes, such as drug and supply procurement, equipment maintenance, routine death audits and selected aspects of clinical case management. We therefore need to caution the reader when extrapolating our results to other RBF settings.…”