Development and psychometric validation of a novel patient survey to assess perceived quality of substance abuse treatment in South Africa

Abstract

BackgroundA hybrid performance measurement system that combines patient-reported outcome data with administrative data has been developed for South African substance abuse treatment services. This paper describes the development and psychometric validation of one component of this system, the South African Addiction Treatment Services Assessment (SAATSA).MethodsFirst, a national steering committee identified five domains and corresponding indicators on which treatment quality should be assessed. A decision was made to develop a patient survey to assess several of these indicators. A stakeholder work group sourced survey items and generated additional items where appropriate. The feasibility and face validity of these items were examined during cognitive response testing with 16 patients. This led to the elimination of several items. Next, we conducted an initial psychometric validation of the SAATSA with 364 patients from residential and outpatient services. Exploratory (EFA) and confirmatory factor analyses (CFA) were conducted to assess the latent structure of the SAATSA. Findings highlighted areas where the SAATSA required revision. Following revision, we conducted another psychometric validation with an additional sample of 285 patients. We used EFA and CFA to assess construct validity and we assessed reliability using Cronbach’s measure of internal consistency.ResultsThe final version of the SAATSA comprised 31 items (rated on a four-point response scale) that correspond to six scales. Four of these scales are patient-reported outcome measures (substance use, quality of life, social connectedness and HIV risk outcomes) that together assess the perceived effectiveness of treatment. The remaining two scales assess patients’ perceptions of access to and quality of care. The models for the final revised scales had good fit and the internal reliability of these scales was good to excellent, with Cronbach’s α ranging from 0.72 to 0.89.ConclusionA lack of adequate measurement tools hampers efforts to improve the quality of substance abuse treatment. Our preliminary evidence suggests that the SAATSA, a novel patient survey that assesses patients’ perceptions of the outcomes and quality of substance abuse treatment, is a psychometrically robust tool that can help fill this void.

Readiness to adopt a performance measurement system for substance abuse treatment: Findings from the Service Quality Measures initiative

“…Alternatively, providers with large caseloads may have been more ready to adopt this system because they expected it to streamline their administrative duties and reduce their paperwork. [6] At the organisational level, greater awareness of the SQM system among staff at the treatment facility, more support from facility leadership for the implementation of the system, more resources available to support adoption of the system, an organisational climate that was more open to change and the adoption of new practices, and greater perceived pressure for change were also related bivariately to high levels of readiness to adopt the SQM performance measurement system. In the multivariate model, the only factors associated with readiness to adopt this system were awareness of the SQM initiative among facility staff and leadership support for the implementation of the system.…”

“…This information collects administrative data on the type of services received and the patient's response to treatment. [7] The developmental process [6,7] and the process evaluation of this system's initial implementation have been reported elsewhere. [8] Despite a robust developmental phase, questions remain about the extent to which treatment facilities will adopt the SQM system.…”

“…[10,11] The items were explored in focus groups with providers and were shown to have good face validity. [6] • Demographic questions. Such questions assessed age and gender.…”

“…the Service Quality Measures (SQM) initiative. This performance measurement system consists of three tools: the SA Community Epidemiology Network on Drug Use (SACENDU) admission form that collects sociodemographic and substance use history information of patients enrolled into substance abuse treatment and is completed by programme administrators; the SA Addiction Treatment Services Assessment (SAATSA), which collects patient-reported data on perceived outcomes, quality and accessibility of treatment and is completed by patients after 2 -3 weeks of treatment; [6] and a discharge form, which counsellors complete after their patients are discharged from or leave treatment. This information collects administrative data on the type of services received and the patient's response to treatment.…”

Providers’ perceptions of the implementation of a performance measurement system for substance abuse treatment: A process evaluation of the Service Quality Measures initiative

“…This developmental process has been described in detail elsewhere. [10][11][12] The finalised system comprises three tools: the SA Community Epidemiology Network on Drug Use (SACENDU)'s admission form that collects sociodemographic and substance use history information and is completed on enrolment into treatment, the SA Addiction Treatment Services Assessment (SAATSA) that accumulates patientreported data on perceived outcomes and quality of services, and a discharge form that gathers data on the type of services received and the patient's response to treatment. To support implementation we developed a toolkit for providers, specifying in detail the protocols to be followed when implementing this system.…”