Blending Qualitative and Computational Linguistics Methods for Fidelity Assessment: Experience with the Familias Unidas Preventive Intervention

Abstract

Careful fidelity monitoring and feedback are critical to implementing effective interventions. A wide range of procedures exist to assess fidelity; most are derived from observational assessments (Schoenwald et al, 2013). However, these fidelity measures are resource intensive for research teams in efficacy/effectiveness trials, and are often unattainable or unmanageable for the host organization to rate when the program is implemented on a large scale. We present a first step towards automated processing of linguistic patterns in fidelity monitoring of a behavioral intervention using an innovative mixed methods approach to fidelity assessment that uses rule-based, computational linguistics to overcome major resource burdens. Data come from an effectiveness trial of the Familias Unidas intervention, an evidence-based, family-centered preventive intervention found to be efficacious in reducing conduct problems, substance use and HIV sexual risk behaviors among Hispanic youth. This computational approach focuses on “joining,” which measures the quality of the working alliance of the facilitator with the family. Quantitative assessments of reliability are provided. Kappa scores between a human rater and a machine rater for the new method for measuring joining reached .83. Early findings suggest that this approach can reduce the high cost of fidelity measurement and the time delay between fidelity assessment and feedback to facilitators; it also has the potential for improving the quality of intervention fidelity ratings.

Factors associated with the implementation of the Familias Unidas intervention in a type 3 translational trial

“…One possibility for moving forward is the use of computational linguistics methods for fidelity assessment, which would automate ratings and increase the ability to monitor and provide timely supervision [40]. Indeed, a proof of concept for the use of computational linguistics has been conducted for Familias Unidas [41]. Specifically, one of the indicators of competence (Bjoining^) was examined in a third of family visits in this trial using a specified decision tree algorithm that categorized transcribed facilitators' Spanish and/or English-language utterances (e.g., use of open-ended questions) as characteristic of low or high joining with families [41].…”

“…Indeed, a proof of concept for the use of computational linguistics has been conducted for Familias Unidas [41]. Specifically, one of the indicators of competence (Bjoining^) was examined in a third of family visits in this trial using a specified decision tree algorithm that categorized transcribed facilitators' Spanish and/or English-language utterances (e.g., use of open-ended questions) as characteristic of low or high joining with families [41]. This algorithm, developed with input from Familias Unidas developers and the lead clinical supervisor, was designed to rate Bwhat/why^questions as more favorable for joining than other types of questions and/or utterances and demonstrated high reliability (kappa=0.83) with human observers.…”

An individually tailored family-centered intervention for pediatric obesity in primary care: study protocol of a randomized type II hybrid effectiveness–implementation trial (Raising Healthy Children study)

“…In addition, we will develop an automated coding system based on an existing, validated automated coding system for motivational interviewing. Automated fidelity coding will be built from previous methodologies developed for coding fidelity to MI (presence of complex reflections, open-ended questions) [ 56 , 64 , 65 ] and family interventions [ 66 , 67 ]. Machine-generated fidelity codes have been found to be reliable with human coding across multiple studies [ 56 , 64 – 66 , 68 , 69 ].…”

Measurement of Sustainment of Prevention Programs and Initiatives: The Sustainment Measurement System Scale

“…While monitoring and feedback are recognized as important for prevention [19,20], much of the relevant science on feedback in health has involved improvement in clinical performance [21][22][23][24][25][26][27]. This includes clinical supervision and use of technology like electronic dashboards in measurement-based quality improvement (MBQI) strategies that monitor patient behavior and clinician activity [28][29][30], while prevention has a more limited history of using computational technologies for monitoring [31][32][33][34]. Such feedback offers the clinician a better understanding of whether they are on course to achieve a successful outcome or need to alter their treatment in order to improve the likelihood of a successful outcome.…”