The TPOCS-self-reported Therapist Intervention Fidelity for Youth (TPOCS-SeRTIFY): A case study of pragmatic measure development

Abstract

Background: Developing pragmatic assessment tools to measure clinician use of evidence-based practices is critical to advancing implementation of evidence-based practices in mental health. This case study details our community-partnered process of developing the Therapy Process Observation Coding Scale-Self-Reported Therapist Intervention Fidelity for Youth (TPOCS-SeRTIFY), a pragmatic, clinician-report instrument to measure cognitive behavioral therapy (CBT) delivery. Approach: We describe a five-step community-partnered development process. Initial goals were to create a self-report instrument that paralleled an existing direct observation measure of clinician delivery of CBT use to facilitate later assessment of measure performance. Cognitive interviews with community clinicians ( n = 6) and consultation with CBT experts ( n = 6) were used to enhance interpretability and usability as part of an iterative refinement process. The instrument was administered to 247 community clinicians along with an established self-reported measure of clinician delivery of CBT and other treatments to assess preliminary psychometric performance. Preliminary psychometrics were promising. Conclusion: Our community-partnered development process showed promising success and can guide future development of pragmatic implementation measures both to facilitate measurement of ongoing implementation efforts and future research aimed at building learning mental health systems. Plain language summary Developing brief, user-friendly, and accurate tools to measure how therapists deliver cognitive behavioral therapy (CBT) in routine practice is important for advancing the reach of CBT into community settings. To date, developing such “pragmatic” measures has been difficult. There is little known about how researchers can best develop these types of assessment tools so that they (1) are easy for clinicians in practice to use and (2) provide valid and useful information about implementation outcomes. As a result, there are few well-validated measures in existence that measure therapist use of CBT that are feasible for use in community practice. This paper contributes to the literature by describing our community-partnered process for developing a measure of therapist use of CBT (Therapy Process Observation Coding Scale -Self-Reported Therapist Intervention Fidelity for Youth; TPOCS-SeRTIFY). This descriptive case study outlines the community-partnered approach we took to develop this measure. This case study will contribute to future research by serving as a guide to others aiming to develop pragmatic implementation measures. In addition, the TPOCS-SeRTIFY is a pragmatic measure of clinician use of CBT that holds promise for its use by both researchers and clinicians to measure the success of CBT implementation efforts.

Concordance of multiple informant assessment of school-based social skills intervention and association with child outcomes: Results from a randomized trial

“…COVID-19 concerns have exacerbated challenges with observational fidelity measurement in schools, given the restrictions the pandemic has placed on researchers’ abilities to physically enter school grounds. The use of provider (i.e., coach- and self-) fidelity ratings allow for the potential benefits of reduced costs and the promotion of a practice-based learning approach for intervention implementers ( Becker-Haimes et al, 2021 ) and may be more feasible as schools continue to grapple with COVID-19 restrictions and changes to school policies regarding external in-person visitors. Additionally, coach- and self-ratings may provide insight into contextual factors such as individual child presentation or provider style, which may be helpful when considering the personalization or individuation of treatment to promote long-term sustainability.…”

“…Clearly, a balancing act is needed to use fidelity measurement that provides meaningful information on intervention recipient outcomes, while accounting for privacy, cost of measurement, and sustainability of implementation monitoring systems beyond a research study. These concerns suggest that pragmatic approaches to measurement, such as provider fidelity ratings, are important, particularly for implementation research that is focused on intervention uptake and sustainment ( Becker-Haimes et al, 2021 ). Determining how to train cost-effective and sustainable fidelity monitors to provide “good enough” ratings that provide meaningful information related to student outcomes is an important next step for implementation research.…”

Implementing Evidence-Based Mental-Health Treatments: Attending to Training, Fidelity, Adaptation, and Context

“…Future research exploring the degree of fidelity needed to achieve good outcomes, adaptations that are most likely to be effective, and how to provide practical, real-time guidance and support to treatment providers could have a substantial impact. Researchers have been investigating more scalable approaches to fidelity assessment and support than the “gold standard” strategy of observing and coding full therapy sessions (e.g., Becker-Haimes et al 2021). Additional work is being done to see how artificial intelligence (Flemotomos et al, 2021; Imel et al, 2019) and other methods can be applied to data collected over the course of treatment to assess and support fidelity (Stirman et al, 2021).…”