Online Training of Community Therapists in Observational Coding of Family Therapy Techniques: Reliability and Accuracy

Hogue, Aaron; Porter, Nicole; Bobek, Molly; MacLean, Alexandra; Bruynesteyn, Lila; Jensen‐Doss, Amanda; Dauber, Sarah; Henderson, Craig E.

doi:10.1007/s10488-021-01152-4

Cited by 10 publications

(6 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While our CPS experts were unlikely to gain more knowledge of the approach itself, it appears that training on the coding system, which includes reading and discussing exemplars and receiving feedback on one's ratings of mock sessions, changed the observers’ sensitivity to the nuances of CPS practice beyond what CPS expertise alone could provide. This finding can inform the development of activities to increase practitioners’ fidelity ( Caron & Dozier, 2022 ; Hogue, Porter, et al, 2022 ).…”

Section: Discussionmentioning

confidence: 85%

“…It may also be that providers’ memories of sessions are generally biased or inaccurate. Several methods have been explored to improve the reliability and accuracy of providers’ self-reports, for example, training providers with exemplars, mock ratings of recorded treatment sessions, and fidelity feedback (e.g., Caron & Dozier, 2022 ; Hogue, MacLean, et al, 2022 ; Hogue, Porter, et al, 2022 ). Other ideas include using structured notetaking during sessions or providing abbreviated recordings to increase the recall accuracy.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Pragmatic fidelity measurement in youth service settings

Wang

Stoll

Eddy

et al. 2023

Implementation Research and Practice

View full text Add to dashboard Cite

Background Fidelity measurement is critical for developing, evaluating, and implementing evidence-based treatments (EBTs). However, traditional fidelity measurement tools are often not feasible for community-based settings. We developed a short fidelity rating form for the Collaborative Problem Solving (CPS) approach from an existing manualized coding system that requires extensive training. We examined the reliability and accuracy of this short form when completed by trained observers, untrained observers, and self-reporting providers to evaluate multiple options for reducing barriers to fidelity measurement in community-based settings. Methods Community-based treatment providers submitted recordings of youth service sessions in which they did, or did not, use CPS. For 60 recordings, we compared short-form fidelity ratings assigned by trained observers and untrained observers to those provided by trained observers on the manualized coding system. For 141 recordings, we compared providers’ self-reported fidelity on the short form to ratings provided by trained observers on the manualized coding system and examined providers’ accuracy as a function of their global fidelity. Results & Conclusions: The short form was reliable and accurate for trained observers. An assigned global integrity score and a calculated average of component scores on the short form, but not component scores themselves, were reliable and accurate for observers who had CPS expertise but no specific training on rating CPS fidelity. When providers self-reported fidelity on the short form, their global integrity score was a reliable estimate of their CPS integrity; however, providers with better CPS fidelity were most accurate in their self-reports. We discuss the costs and benefits of these more pragmatic fidelity measurement options in community-based settings.

show abstract

Section: Discussionmentioning

confidence: 85%

Section: Discussionmentioning

confidence: 99%

Pragmatic fidelity measurement in youth service settings

Wang

Stoll

Eddy

et al. 2023

Implementation Research and Practice

View full text Add to dashboard Cite

show abstract

“…Therefore, meaningful conclusions from this trial were not affected by somewhat higher than anticipated ICCs. Like other studies, we relied on self-reported data to measure clinician and beneficiary constructs [ 43 , 44 ]. We took several steps to mitigate this bias, including reminding the clinicians that their responses would not be shared with others and using standardized data collection tools.…”

Section: Discussionmentioning

confidence: 99%

Results of a multi-site pragmatic hybrid type 3 cluster randomized trial comparing level of facilitation while implementing an intervention in community-dwelling disabled and older adults in a Medicaid waiver

et al. 2022

View full text Add to dashboard Cite

Background Evidence-based interventions that optimize physical function for disabled and older adults living in the community who have difficulty with daily living tasks are available. However, uptake has been limited, particularly in resource-constrained (Medicaid) settings. Facilitation may be an effective implementation strategy. This study’s aim was to compare internal facilitation (IF) versus IF and external facilitation (EF) on adoption and sustainability of an intervention in a Medicaid home and community-based waiver. Methods In a hybrid type 3 trial, waiver sites (N = 18) were randomly assigned to implement the intervention using a bundle of strategies with either IF or IF and EF. Adoption and sustainability were assessed via Stages of Implementation Completion (SIC) for each site. Clinician attitudes toward evidence-based practice and self-efficacy were evaluated among 539 registered nurses, social workers, and occupational therapists. Medicaid beneficiary outcomes of activities of daily living, depression, pain, falls, emergency department visits, and hospitalizations were evaluated in a sample of N = 7030 as reflected by electronic health records data of the Medicaid waiver program. Linear mixed-effects models were used to compare outcomes between trial arms while accounting for cluster-randomized design. Results The mean SIC scores were 72.22 (standard deviation [SD] = 16.98) in the IF arm (9 sites) and 61.33 (SD = 19.29) in the IF + EF arm (9 sites). The difference was not statistically significant but corresponded to the medium clinically important effect size Cohen’s d = 0.60. Clinician implementation outcomes of attitudes and self-efficacy did not differ by trial arm. Beneficiary depression was reduced significantly in the IF + EF arm compared to the IF arm (p = .04, 95% confidence interval for the difference [0.01, 0.24]). No differences between trial arms were found for other beneficiary outcomes. Conclusions Level of facilitation did not enhance capacity for adoption and sustainability of an evidence-based intervention in a Medicaid setting that cares for disabled and older adults. Improved beneficiary depression favored use of IF and EF compared to IF alone, and no differences were found for other outcomes. These findings also suggest level of facilitation may not have impacted beneficiary outcomes. Trial registration ClinicalTrials.gov, NCT03634033; date registered August 16, 2018.

show abstract

“…Particularly elusive are measurement methods that are both efficient (i.e., feasible) and effective (i.e., rigorously evaluated, psychometrically sound; Schoenwald et al, 2011 ). Observational methods have been a gold standard ( Stirman, 2020 ), but their feasibility in community mental health settings is presumed limited due to resource constraints ( Hogue et al, 2021 ). Until recently, information about fidelity measurement appeared primarily in published treatment efficacy or effectiveness studies and lacked details needed to estimate the associated resource requirements (Schoenwald & Garland, 2013).…”

Section: Introductionmentioning

confidence: 99%

“…However, for clinician self-ratings to be accurate and reliable, training may be required, as the concordance of therapist and observer ratings of EBI components has typically been low ( Caron, et al., 2020 ). In a recent study, an innovative online training system was used to enable community-based therapists to practice observational coding of family therapy techniques in mock video vignettes ( Hogue et al, 2021 ). There were promising training effects on the reliability of therapist ratings—but less so for accuracy—which led the investigators to conclude that stronger training effects would be needed for a meaningful impact on EBI implementation.…”

Section: Introductionmentioning

confidence: 99%

Improving the feasibility of fidelity measurement for community-based quality assurance: Partial- versus full-session observations of supervisor adherence and competence

Chapman

Alley

Schoenwald

2022

Implementation Research and Practice

View full text Add to dashboard Cite

Background Clinical supervision is a common quality assurance method for supporting the implementation and sustainment of evidence-based interventions (EBIs) in community mental health settings. However, assessing and supporting supervisor fidelity requires efficient and effective measurement methods. This study evaluated two observational coding approaches that are potentially more efficient than coding full sessions: a randomly selected 15-min segment and the first case discussion of the session. Method Data were leveraged from a randomized trial of an Audit and Feedback (A&F) intervention for supervisor Adherence and Competence. Supervisors ( N = 57) recorded and uploaded weekly group supervision sessions for 7 months, with one session observationally coded each month ( N = 374). Of the coded sessions, one was randomly selected for each supervisor, and a random 15-min segment was coded. Additionally, the first case discussion was coded for the full sample of sessions. Results Across all models (and controlling for the proportion of the session covered by the partial observation), Adherence and Competence scores from partial observations were positively and significantly associated with scores from full sessions. In all cases, partial observations were most accurate when the level of Adherence and Competence was moderate. At lower levels, partial observations were underestimates, and at higher levels, they were overestimates. Conclusions The results suggest that efficient observational measurement can be achieved while retaining a general level of measurement effectiveness. Practically, first-case discussions are easier to implement, whereas 15-min segments have fewer potential threats to validity. Evaluation of resource requirements is needed, along with determining whether A&F effects are retained if feedback is based on partial observations. Nevertheless, more efficient observational coding could increase the feasibility of routine fidelity monitoring and quality assurance strategies, including A&F, which ultimately could support the implementation and sustainment of effective supervision practices and EBIs in community practice settings. Plain Language Summary: When delivering evidence-based mental health interventions in community-based practice settings, a common quality assurance method is clinical supervision. To support supervisors, assessment methods are needed, and those methods need to be both efficient and effective. Ideally, supervision sessions would be recorded, and trained coders would rate the supervisor’s use of specific strategies. In most settings, though, this requires too many resources. The present study evaluated a more efficient approach. The data came from an existing randomized trial of an Audit and Feedback intervention for enhancing supervisor Adherence and Competence. This included 57 supervisors and 374 sessions across seven months of monitoring. Instead of rating full supervision sessions, a more efficient approach was to have coders rate partial sessions. Two types of partial observations were considered: a randomly selected 15-minute segment of the session and the first case discussion of the session. The aim was to see if partial observations and full observations led to similar conclusions about Adherence and Competence. In all cases, they did. The scores were most similar for sessions with moderate levels of Adherence and Competence. If Adherence and Competence were low, partial observations were underestimates, but if they were high, partial observations were overestimates. Observing partial sessions is more efficient, but in terms of accuracy, the benefits and limitations should be evaluated in light of how the scores will be used. Additionally, future research should consider whether Audit and Feedback interventions have the same effect if feedback is based on observations of partial sessions.

show abstract

Online Training of Community Therapists in Observational Coding of Family Therapy Techniques: Reliability and Accuracy

Cited by 10 publications

References 47 publications

Pragmatic fidelity measurement in youth service settings

Pragmatic fidelity measurement in youth service settings

Results of a multi-site pragmatic hybrid type 3 cluster randomized trial comparing level of facilitation while implementing an intervention in community-dwelling disabled and older adults in a Medicaid waiver

Improving the feasibility of fidelity measurement for community-based quality assurance: Partial- versus full-session observations of supervisor adherence and competence

Contact Info

Product

Resources

About