Responding to policy makers’ evaluation needs: combining experimental and quasi-experimental approaches to estimate the impact of performance based financing in Burkina Faso

Abstract

BackgroundThe last two decades have seen a growing recognition of the need to expand the impact evaluation toolbox from an exclusive focus on randomized controlled trials to including quasi-experimental approaches. This appears to be particularly relevant when evaluation complex health interventions embedded in real-life settings often characterized by multiple research interests, limited researcher control, concurrently implemented policies and interventions, and other internal validity-threatening circumstances. To date, however, most studies described in the literature have employed either an exclusive experimental or an exclusive quasi-experimental approach.MethodsThis paper presents the case of a study design exploiting the respective advantages of both approaches by combining experimental and quasi-experimental elements to evaluate the impact of a Performance-Based Financing (PBF) intervention in Burkina Faso. Specifically, the study employed a quasi-experimental design (pretest-posttest with comparison) with a nested experimental component (randomized controlled trial). A difference-in-differences approach was used as the main analytical strategy.DiscussionWe aim to illustrate a way to reconcile scientific and pragmatic concerns to generate policy-relevant evidence on the intervention’s impact, which is methodologically rigorous in its identification strategy but also considerate of the context within which the intervention took place. In particular, we highlight how we formulated our research questions, ultimately leading our design choices, on the basis of the knowledge needs expressed by the policy and implementing stakeholders. We discuss methodological weaknesses of the design arising from contextual constraints and the accommodation of various interests, and how we worked ex-post to address them to the best extent possible to ensure maximal accuracy and credibility of our findings. We hope that our case may be inspirational for other researchers wishing to undertake research in settings where field circumstances do not appear to be ideal for an impact evaluation.Trial registrationRegistered with RIDIE (RIDIE-STUDY-ID-54412a964bce8) on 10/17/2014.

Do Targeted User Fee Exemptions Reach the Ultra-Poor and Increase their Healthcare Utilisation? A Panel Study from Burkina Faso

“…Given the very low healthcare utilisation among the ultra-poor, the government of Burkina Faso adapted various policies to explicitly exempt them from paying for healthcare services at the primary level; first in the context of the Bamako Initiative but also in consecutive years [ 17 , 18 , 19 , 20 ]. The current law on Universal Health Insurance ( loi sur l’assurance maladie universelle ) adopted in September 2015, also recognised the liability of the state to fully pay the user fees of the ultra-poor.…”

“…In addition to the many policy attempts, the Burkinabè government implemented several exemption pilot programmes on a smaller scale [ 21 , 22 ]. In 2014, Burkina Faso tested performance-based financing (PBF) [ 18 , 19 ] in combination with targeted user fee exemptions [ 20 ] on a larger scale in eight health districts (Diébougou, Batié, Kongoussi, Kaya, Ouargaye, Tenkodogo, Gourcy, Ouahigouya). Community-based targeting (CBT) was used to identify up to 20% of all individuals residing in the district as extremely poor.…”

"It Depends on What They Experience in Each Health Facility. Some Are Satisfied, Others Are Not." A MixedMethods Exploration of Health Workers’ Attitudes Towards Performance-Based Financing in Burkina Faso

“…The intervention and its background and context are described in detail elsewhere. 32,33 Although the primary objective was to improve utilization and quality of maternal and child health services, the intervention effectively included a broad range of primary-and secondary-level services, including also curative care, TB, and HIV services. In brief, health facilities signed contracts with the MoH stipulating the services purchased by PBF, a comprehensive list of quality indicators, and payment modalities.…”

“…This also included a calculator to determine bonus amounts for individual staff members, based on 5 criteria. In some health facilities, following a randomization process in the context of an impact evaluation, 32 the standard PBF was further complemented with measures intended to increase equity in impact.…”

“…Quantitative data were collected in the context of the abovementioned impact evaluation. The study design and sampling procedures are described in detail in De Allegri et al 32 In brief, the study included all 396 primary-level healthcare facilities in all 12 purposely selected intervention health districts that newly received PBF in 2014. In line with the specific objectives set for the study presented in this paper, we only used endline data, collected between April and June 2017, approximately 3 years after the introduction of PBF.…”