A Mixed-Methods Evaluation of the Implementation of a Performance Measurement System for South Africa’s Substance Use Treatment Services

Abstract

Objective:Minimal knowledge exists on the factors that affect implementation of performance measurement systems, particularly in low- and middle-income countries (LMICs). To address this, we describe the implementation of a performance measurement system for South Africa’s substance abuse treatment services known as the Service Quality Measures (SQM) initiative.Method:We conducted a mixed-methods evaluation of system implementation. We surveyed 81 providers about the extent of system implementation within their agencies and the factors that facilitated implementation. We conducted 26 in-depth interviews of providers’ perceived barriers and facilitators to implementation.Results:The overall penetration of this system was high. Almost all providers viewed the system as feasible to implement, acceptable, appropriate for use in their context, and useful for guiding service improvements. However, the extent of implementation varied significantly across sites (p < .05). Leadership support (p < .05) was associated with increased implementation in multivariable analyses. Providers reflected that high rates of patient attrition, variability in willingness to implement the system, and limited capacity for interpreting performance feedback affected the extent of system implementation.Conclusions:It is feasible to implement a performance measurement system in LMICs if the system is acceptable, appropriate, and useful to providers. To ensure the utility of this system for treatment service strengthening, system implementation must be optimized. Efforts to enhance target population coverage, strengthen leadership support for performance measurement, and build capacity for performance feedback utilization may enhance the implementation of this performance measurement system.Objectifs :Peu de connaissance existe sur les facteurs influençant l’implantation d’un système de mesure du rendement, particulièrement dans les pays à revenu faible et intermédiaire (PRFI). Pour aborder cette question, nous décrivons l’implantation d’un système de mesure du rendement des services de traitement en toxicomanie d’Afrique du Sud, connu sous le nom du projet de Mesures de la qualité du service (MQS).Méthode :Nous avons mené une évaluation d’implantation du système à l’aide d’une méthode mixte. Nous avons interrogé 81 pourvoyeurs de services sur l’ampleur de l’implantation du système dans leur agence et les facteurs qui ont facilité l’implantation. Nous avons mené 26 entrevues approfondies sur les perceptions des pourvoyeurs concernant les obstacles et les facilitateurs de l’implantation.Résultats :L’appropriation générale de ce système était élevée. Presque tous les pourvoyeurs voyaient la faisabilité d’implantation du système et le considéraient comme étant acceptable, approprié à leur contexte et utile pour guider l’amélioration des services. Cependant, l’ampleur de l’implantation variait de façon significative à travers les sites (p < ,05). Le soutien de la direction (p < ,05) était associé à une implantation plus importante dans les analyses ...

South African service users’ perceptions of patient‐reported outcome and experience measures for adolescent substance use treatment: A qualitative study

“…Since 2014, this system has been implemented in adult SUD services in the Western Cape. Implementation involves the completion of three administrative forms for all adult service users entering treatment: (i) the South African Community Epidemiology Network on Drug Use (SACENDU) admission form that providers complete when service users enter treatment , (ii) the South African Addiction Treatment Services Assessment (SAATSA) that service users complete towards the end of the program, and (iii) a standardised discharge form completed by providers within 30 days of a service user ending treatment .…”

“…While the SQM system yields information that providers find useful for guiding service improvements, scaling up system implementation has been hampered by provider concerns about its comprehensiveness and reach, specifically the proportion and representativeness of service users who provide feedback . A key constraint has been the exclusion of adolescent service users who comprise approximately a fifth of all SUD treatment users .…”

“…Implementation involves the completion of three administrative forms for all adult service users entering treatment: (i) the South African Community Epidemiology Network on Drug Use (SACENDU) admission form that providers complete when service users enter treatment [6], (ii) the South African Addiction Treatment Services Assessment (SAATSA) that service users complete towards the end of the program, and (iii) a standardised discharge form completed by providers within 30 days of a service user ending treatment [10,11]. While the SQM system yields information that providers find useful for guiding service improvements, scaling up system implementation has been hampered by provider concerns about its comprehensiveness and reach, specifically the proportion and representativeness of service users who provide feedback [10]. A key constraint has been the exclusion of adolescent service users who comprise approximately a fifth of all SUD treatment users [12].…”

“…However, the impact of these efforts has been difficult to assess due to limited monitoring of service quality and outcomes. To address this gap, we developed a performance measurement system for South Africa's adult SUD services, known as the Service Quality Measures (SQM) initiative, that assesses the effectiveness, efficiency, access to and quality of SUD treatment . Since 2014, this system has been implemented in adult SUD services in the Western Cape.…”

System Performance Measurement: Implications for Service Planning

“…Urbanoski & Inglis (2019) report the results of a scoping review on performance measurement in mental health and addiction treatment systems, concluding that a wide variety of indicators is available to those designing a measurement system. Myers et al (2019) extend this review to cover performance measurement systems for low- and middle-income countries (LMICs) such as South Africa, showing that it is feasible to implement performance measurement systems in LMICs if the system is acceptable, appropriate, and useful. Finally, Montanari et al (2019) demonstrate the utility of implementing a common treatment demand indicator in Europe, across 30 countries.…”