Psychometric performance of the Mental Health Implementation Science Tools (mhIST) across six low- and middle-income countries

Abstract

Background Existing implementation measures developed in high-income countries may have limited appropriateness for use within low- and middle-income countries (LMIC). In response, researchers at Johns Hopkins University began developing the Mental Health Implementation Science Tools (mhIST) in 2013 to assess priority implementation determinants and outcomes across four key stakeholder groups—consumers, providers, organization leaders, and policy makers—with dedicated versions of scales for each group. These were field tested and refined in several contexts, and criterion validity was established in Ukraine. The Consumer and Provider mhIST have since grown in popularity in mental health research, outpacing psychometric evaluation. Our objective was to establish the cross-context psychometric properties of these versions and inform future revisions. Methods We compiled secondary data from seven studies across six LMIC—Colombia, Myanmar, Pakistan, Thailand, Ukraine, and Zambia—to evaluate the psychometric performance of the Consumer and Provider mhIST. We used exploratory factor analysis to identify dimensionality, factor structure, and item loadings for each scale within each stakeholder version. We also used alignment analysis (i.e., multi-group confirmatory factor analysis) to estimate measurement invariance and differential item functioning of the Consumer scales across the six countries. Results All but one scale within the Provider and Consumer versions had Cronbach’s alpha greater than 0.8. Exploratory factor analysis indicated most scales were multidimensional, with factors generally aligning with a priori subscales for the Provider version; the Consumer version has no predefined subscales. Alignment analysis of the Consumer mhIST indicated a range of measurement invariance for scales across settings (R2 0.46 to 0.77). Several items were identified for potential revision due to participant nonresponse or low or cross- factor loadings. We found only one item, which asked consumers whether their intervention provider was available when needed, to have differential item functioning in both intercept and loading. Conclusion We provide evidence that the Consumer and Provider versions of the mhIST are internally valid and reliable across diverse contexts and stakeholder groups for mental health research in LMIC. We recommend the instrument be revised based on these analyses and future research examine instrument utility by linking measurement to other outcomes of interest.

Implementation measurement in global mental health: Results from a modified Delphi panel and investigator survey

“…This guidance should facilitate rigorous and replicable implementation research in an area of high need, though it is not intended to be prescriptive, and local investigators are encouraged to adapt and apply the guidance only where it is useful. Moving forward, as the quantity and quality of implementation measures designed for use in for diverse global contexts increase (Aldridge et al, 2022), the standards for measure adaptation and validation may also shift. Less emphasis may be placed on establishing measure validity for the sake of scientific rigor, with a corresponding increased emphasis on measure pragmatic qualities and capacity to inform real-world health service delivery.…”

Strategies to improve the implementation and effectiveness of community-based psychosocial support interventions for displaced, migrant and host community women in Latin America

“…Semi-structured interviews were designed to capture information about the following implementation outcomes: acceptability, relevance/appropriateness, feasibility, reach and accessibility, effectiveness, safety, implementation (including barriers and facilitators) and sustainability. Questions were informed by the Johns Hopkins Dissemination and Implementation Science Measure (Haroz and Murray, 2018 ; Aldridge et al, 2022 ). Interviews lasted approximately 45 min and were conducted by a member of the research team in Spanish either in person (48.6%) or by phone (51.4%).…”

Development and Piloting of a Mental Health Prevention and Referral Program for Veterans and Their Families in Ukraine

“…At the 1-month follow-up, participants also completed the Mental Health Implementation Science Tools Consumer version, 25 , 26 an implementation research measure that assesses consumer perceptions of program adoptability (4 items), acceptability (9 items), appropriateness (6 items), feasibility (8 items), and accessibility (8 items). Participants responded to statements related to each implementation domain, such as whether they would use CPSS in the future if needed, using Likert-style response options from 0 to 3 (“no,” “somewhat no,” “somewhat yes,” and “yes”).…”