Real-Time Assessments of the Strength of Program Implementation for Community Case Management of Childhood Illness: Validation of a Mobile Phone-Based Method in Malawi

Abstract

Health surveillance assistants (HSAs) in Malawi have provided community case management (CCM) since 2008; however, program monitoring remains challenging. Mobile technology holds the potential to improve data, but rigorous assessments are few. This study tested the validity of collecting CCM implementation strength indicators through mobile phone interviews with HSAs. This validation study compared mobile phone interviews with information obtained through inspection visits. Sensitivity and specificity were measured to determine validity. Using mobile phones to interview HSAs on CCM implementation strength indicators produces accurate information. For deployment, training, and medicine stocks, the specificity and sensitivity of the results were excellent (> 90%). The sensitivity and specificity of this method for drug stock-outs, supervision, and mentoring were lower but with a few exceptions, still above 80%. This study provided a rigorous assessment of the accuracy of implementation strength data collected through mobile technologies and is an important step forward for evaluation of public health programs.

Measuring Implementation Strength for Integrated Community Case Management in Malawi: Results from a National Cell Phone Census

“…The quality of data produced by Health Management Information Systems—even when these systems include relevant data points—is often unknown and records are universally suspected to be incomplete. 3 New and complementary methods are needed to produce up-to-the-moment snapshots of implementation strength for use in improving health service provision, calibrating routine monitoring systems, and interpreting changes (or lack of change) in measures of program outcome and impact.…”

Testing the validity and feasibility of using a mobile phone-based method to assess the strength of implementation of family planning programs in Malawi

“…Even so, the IC kept more records of these indicators than the HSAs and CBDAs. Similar studies, such as Hazel et al, demonstrated higher sensitivity and specificity largely because they evaluated a very specific program that had been recently implemented with clear, measurable components [14]. The ISA used for this study was much broader, as it aimed to assess multiple FP programs implementing a wider set of FP practices.…”

“…The rapid increase in mobile phone ownership and network availability in lower income countries could offer an alternative, cost-effective avenue to collect data for a better understanding of program implementation. However, there is still uncertainty about the validity and feasibility of remote data collection in lower income countries, especially among health workers [14,31]. This study tests whether using this m-Health method can produce valid IS data and can this be a feasible alternative to traditional data collection methods.…”

“…A suite of tools is being developed to assist in these types of evaluations in LMICs by the Institute of International Programs (IIP) [13]. The implementation strength assessment (ISA) is one such tool that is designed to rapidly measure the quantity or dose of a program delivered to its target population and has been applied in a number of contexts [14][15][16][17]. For instance, the ISA was used in Ethiopia to show that integrated community case management of childhood illness can be implemented at scale, as nearly all health extension workers surveyed in Ethiopia were trained, supervised, and had the commodities they needed [15].…”

New Horizons in Design Science: Broadening the Research Agenda

“…In the last decade, the diagnosis and treatment of childhood illness in developing countries has received immense attention (Hazel et al, 2015). The underlying rationale for this increased awareness may stem from the establishment of the eight Millennium Development Goals by the United Nations in 2001, more specifically the fourth goal which focuses on reducing under-five child mortality rates in developing countries.…”