Applying the Realist Evaluation Approach to the Complex Process of Policy Implementation—The Case of the User Fee Exemption Policy for Cesarean Section in Benin

Abstract

Realist evaluation is making inroads in the field of health policy and systems research to a large extent because of its good fit with complex issues. Until now, most realist studies focused on evaluating interventions or projects related to health care delivery, organization of health services, education, management, and leadership of health workers in high income countries. With this paper, we apply the realist approach to the study of national health policy implementation in a low resource country. We use the case of the user fee exemption policy for cesarean section in Benin, which we followed up from 2009 to 2018. We report on how realist evaluation can be applied for policy implementation research. We illustrate how we developed the initial programme theory—the starting point of any realist evaluation -, how we designed the study and data collection tools, and how we analyzed the data. For each step, we present current good practices, how we adapted them when needed, the challenges and the lessons learned. We report also on how the dynamic interactions between the central level (the national implementing agency) and the peripheral level (an implementing hospital) shaped the policy implementation. We found that at central level, availability of resources for a given policy is constantly challenged in the competitive national resource allocation arena. Key factors include the political power and the legitimacy of the group supporting the policy. These are influenced by the policy implementation structure, how the actual outputs of the implementation align with promises of the group supporting the policy and consequently how these outputs, the policy and its promoters are perceived by the community. We found that the service providers are key to the implementation, and that they are constrained or influenced by the dependability of the funding, their autonomy, their personal background, and the accountability arrangements. This study can inform the design and implementation of national health policies that involve interactions between central and operational level in other low-income countries.

How do oilcloth sessions work? A realist evaluation approach to exploring ripple effects in an implementation strategy

“…The use of a realist evaluation approach is particularly applicable when exploring how complex systems, such as implementation strategies, work, and to identify unintended outcomes that may interfere with achieving the intended outcomes. Although identifying context is a key element of the realist evaluation approach, defining the context and separating mechanisms from the context is difficult because any changes among the participants, the organizational context or the overall implementation can affect the implementation outcomes (Dossou et al ., 2021). Based on this perspective, an implementation strategy will not have the same outcome in every target group or in any institutional context or infrastructure, and changes in a particular context may trigger other mechanisms than identified in a study.…”

“…Furthermore, the study highlights important contextual factors relevant for effective implementation of organizational changes, such as organizational uncertainty and time. The importance of context is widely recognized in implementation and evaluation research (Dossou et al ., 2021; Poland et al ., 2008), however, the realist evaluation approach in this study contributes with a focus on context that transcends the implementation strategy by revealing unintended ripple effects caused by mechanisms triggered by the existing contexts.…”

Understanding sustainable outcomes in international development: Towards a realist evaluation framework

“…There is a growing literature using realist evaluation in the field of international development. Recent examples in the field of health includeMarchal et al (2010);Graham et al (2018); andDossou et al (2021). These studies identify the different ways in which programmes work for different people.5 This addresses the critique of external validity that applies to RCTs and other evaluation approaches (whereby the results from an evaluation cannot necessarily be generalised to other contexts).…”