Multiple and mixed methods in formative evaluation: Is more better? Reflections from a South African study

Abstract

BackgroundFormative programme evaluations assess intervention implementation processes, and are seen widely as a way of unlocking the ‘black box’ of any programme in order to explore and understand why a programme functions as it does. However, few critical assessments of the methods used in such evaluations are available, and there are especially few that reflect on how well the evaluation achieved its objectives. This paper describes a formative evaluation of a community-based lay health worker programme for TB and HIV/AIDS clients across three low-income communities in South Africa. It assesses each of the methods used in relation to the evaluation objectives, and offers suggestions on ways of optimising the use of multiple, mixed-methods within formative evaluations of complex health system interventions.MethodsThe evaluation’s qualitative methods comprised interviews, focus groups, observations and diary keeping. Quantitative methods included a time-and-motion study of the lay health workers’ scope of practice and a client survey. The authors conceptualised and conducted the evaluation, and through iterative discussions, assessed the methods used and their results.ResultsOverall, the evaluation highlighted programme issues and insights beyond the reach of traditional single methods evaluations. The strengths of the multiple, mixed-methods in this evaluation included a detailed description and nuanced understanding of the programme and its implementation, and triangulation of the perspectives and experiences of clients, lay health workers, and programme managers. However, the use of multiple methods needs to be carefully planned and implemented as this approach can overstretch the logistic and analytic resources of an evaluation.ConclusionsFor complex interventions, formative evaluation designs including multiple qualitative and quantitative methods hold distinct advantages over single method evaluations. However, their value is not in the number of methods used, but in how each method matches the evaluation questions and the scientific integrity with which the methods are selected and implemented.

Using a mHealth system to recall and refer existing clients and refer community members with health concerns to primary healthcare facilities in South Africa: a feasibility study

“…Our mixed-methods design provided a more nuanced understanding [52,53] of the implementation successes and challenges of the mHealth system than could have been achieved through one method alone, with the qualitative findings being used to understand and corroborate the quantitative findings [28,29,54]. We enhanced the trustworthiness of our qualitative data through including one of the NGO supervisors and research participants [PH] as coauthor.…”

“…We used both quantitative and qualitative methods of data collection and analyses to assess the feasibility of the multi-component mHealth system [27,28]. This required a research team experienced in both approaches [29]. We used an embedded design in which the primary focus was on the quantitative dataset.…”

“…They perform a variety of tasks, such as medication administration, child health surveys, referring people with illness symptoms to facilities, and recalling clients to for various reasons, for instance, to receive tests results or medication [10]. LHWs are also key to ensuring that clients are not lost to follow-up [10,11]. Despite these important functions being performed by LHWs, their turnover is high [12], resulting in a need for constant retraining of staff [13].…”

A mixed methods quality improvement study to implement nurse practitioner roles and improve care for residents in long-term care facilities

“…A mixed methods quality improvement study was conducted [55,56]. The Standards for Quality Improvement Reporting Excellence (SQUIRE 2.0) reporting guidelines were used with attention paid to describing the local context of implementation [57,58].…”

Trying to bridge the worlds of home visitation and child welfare: Lessons learned from a formative evaluation

“…This means that the potential effectiveness is largely speculative. Formative evaluation allows other researchers to see inside the “black box” of complex implementation efforts (Odendaal, Atkins, & Lewin, 2016). This form of evaluation is designed to reveal potential facilitators and barriers to implementation that occur as a project progresses (Stetler et al, 2006).…”