Measuring implementation in global mental health: validation of a pragmatic implementation science measure in eastern Ukraine using an experimental vignette design

Abstract

Background There is mounting evidence supporting the effectiveness of task-shifted mental health interventions in low- and middle-income countries (LMIC). However, there has been limited systematic scale-up or sustainability of these programs, indicating a need to study implementation. One barrier to progress is a lack of locally relevant and valid implementation measures. We adapted an existing brief dissemination and implementation (D&I) measure which includes scales for acceptability, appropriateness, feasibility and accessibility for local use and studied its validity and reliability among a sample of consumers in Ukraine. Methods Local qualitative data informed adaptation of the measure and development of vignettes to test the reliability and validity. Participants were veterans and internally displaced persons (IDPs) recruited as part of a separate validity study of adapted mental health instruments. We examined internal consistency reliability, test-retest reliability, and construct and criterion validity for each scale on the measure. We randomly assigned half the participants to respond to a vignette depicting existing local psychiatric services which we knew were not well regarded, while the other half was randomized to a vignette describing a potentially more well-implemented mental health service. Criterion validity was assessed by comparing scores on each scale by vignette and by overall summary ratings of the programs described in the vignettes. Results N = 169 participated in the qualitative study and N = 153 participated in the validity study. Qualitative findings suggested the addition of several items to the measure and indicated the importance of addressing professionalism/competency of providers in both the scales and the vignettes. Internal consistency reliabilities ranged from α = 0.85 for feasibility to α = 0.91 for appropriateness . Test-rest reliabilities were acceptable to good for all scales ( rho: 0.61–0.79). All scales demonstrated substantial and significant differences in average scores by vignette assignment ( ORs: 2.21–5.6) and overall ratings ( ORs: 5.1–14.47), supporting criterion validity. Conclusions This study represents an innovative mixed-methods approach to testing an implementation science measure in contexts outside the United States. Results support the reliability and validity of most scales for consumers in Ukraine. Challenges included large amounts of missing data due to participants’ difficulties responding to questions about a hypothetical program. Electronic supplementary material The online version of this article (10.1186/s12913-019-4097-y) contains supplementary material, which is available to authorized...

Advancing successful implementation of task-shifted mental health care in low-resource settings (BASIC): protocol for a stepped wedge cluster randomized trial

“…At the counselor-level, we capture demographics (e.g., age, sex), background characteristics (e.g., years in role, education), any training in, or past experience with mental health, and participation in supervision (e.g., dose). When possible, we also assess other implementation constructs using existing measures developed for use in low-resource contexts [51] and/or standardized measures reviewed and translated for appropriateness to assess intervention acceptability, feasibility [52], and perceived intervention effectiveness [51]. For theorized important constructs for which existing scales are not available or are not relevant (i.e., behavioral control, behavioral intentions, self-efficacy, appropriateness, innovation-values fit), we followed measure construction guidelines from the Theoretical Domains Framework [53,54].…”

Project Khanya: a randomized, hybrid effectiveness-implementation trial of a peer-delivered behavioral intervention for ART adherence and substance use in Cape Town, South Africa

“…• Qualitative interviews with providers and organizational leadership based on RE-AIM to assess perceptions and likelihood of uptake following the trial; longer-term adoption to be assessed in the subsequent trial following from this pilot study Implementation Feasibility • Pragmatic implementation science measure based on RE-AIM (14-item feasibility subscale) [31] • % assigned to the intervention who agree to enroll • Qualitative interviews with patients structured based on RE-AIM Acceptability…”

“…Feasibility, defined as the fit or utility of the intervention or suitability and practicability for this setting [30], is also being assessed using two quantitative assessments: (1) the 14-item feasibility subscale of the AMHR pragmatic assessment measure based on RE-AIM [31] and (2) the percentage assigned to the intervention who agree to enroll in the intervention. The feasibility subscale also shows good psychometric properties (internal consistency ∝ = .85; test-retest reliability rho = .79).…”

“…Descriptive statistics will be generated to evaluate implementation outcomes. Total scores from the acceptability and feasibility subscales of the AMHR pragmatic, implementation assessment [31] will be created; these scores, alongside data obtained on provider fidelity to the intervention and patient participation and retention, will first be evaluated descriptively. The means and standard deviation of provider fidelity at each session will be calculated.…”

Lay Counselor Perspectives of Providing a Child-Focused Mental Health Intervention for Children: Task-Shifting in the Education and Health Sectors in Kenya

“…While qualitative approaches provide rich data about counselor experiences, they require intensive resources and time that some mental health implementation efforts may not have and may limit comparison across studies. Only recently have quantitative scales been developed to assess these constructs (28,29), with their use potentially allowing for expanded research in this area.…”