Automatic rating of therapist facilitative interpersonal skills in text: A natural language processing application

Abstract

BackgroundWhile message-based therapy has been shown to be effective in treating a range of mood disorders, it is critical to ensure that providers are meeting a consistently high standard of care over this medium. One recently developed measure of messaging quality–The Facilitative Interpersonal Skills Task for Text (FIS-T)–provides estimates of therapists’ demonstrated ability to convey psychotherapy's common factors (e.g., hopefulness, warmth, persuasiveness) over text. However, the FIS-T's scoring procedure relies on trained human coders to manually code responses, thereby rendering the FIS-T an unscalable quality control tool for large messaging therapy platforms.ObjectiveIn the present study, researchers developed two algorithms to automatically score therapist performance on the FIS-T task.MethodsThe FIS-T was administered to 978 messaging therapists, whose responses were then manually scored by a trained team of raters. Two machine learning algorithms were then trained on task-taker messages and coder scores: a support vector regressor (SVR) and a transformer-based neural network (DistilBERT).ResultsThe DistilBERT model had superior performance on the prediction task while providing a distribution of ratings that was more closely aligned with those of human raters, versus SVR. Specifically, the DistilBERT model was able to explain 58.8% of the variance (R2 = 0.588) in human-derived ratings and realized a prediction mean absolute error of 0.134 on a 1–5 scale.ConclusionsAlgorithms can be effectively used to ensure that digital providers meet a consistently high standard of interactions in the course of messaging therapy. Natural language processing can be applied to develop new quality assurance systems in message-based digital psychotherapy.

A natural language processing approach reveals first-person pronoun usage and non-fluency as markers of therapeutic alliance in psychotherapy

“…Another common language technique that has been used to characterize speaking style of a patient or therapist in the previous literature is n-grams (a single word or short, multi-word phrases in length of n). 11 , 13 , 47 N-gram based models characterize language use as the probability of speaking any word at the present instance, given the preceding word or words. We first encoded a sequence of all words spoken by each speaker in the session with Part-Of-Speech tagging, a tokenization method that assigns each word a syntactic label (e.g.…”

“…Much of existing computational work unfortunately fails to provide such interpretability. For example, several studies using natural language processing (NLP), a subset of artificial intelligence that learns data structure from human language, have revealed algorithms that could predict patient-rated alliance, 11 therapist skills, 12 , 13 and therapeutic rupture events from session transcripts. 14 However, these algorithms are generally trained from the entire set of sentences uttered by a speaker and often provide high dimensional predictive features that are hard to interpret.…”

A review of the facilitative interpersonal skills performance task and rating method.

“…These alternate forms include different languages such as Norwegian, German, Dutch, Danish, Portuguese, and Chinese (Anderson, Finkelstein, & Horvath, 2020; van Thiel et al, 2021) and different populations, such as child therapy (Bate & Tsakas, 2022). Similarly, the success of the rating based method in response to brief challenging situations has led to the development of an FIS-self-report measure (Anderson, Finkelstein, & Horvath, 2020), and FIS-IS in-session method (for rating extracts of actual therapy situations; Uhlin, 2011), FIS-text for evaluating text responses and adds the prospect of using machine learning for evaluation (Anderson, Finkelstein, & Horvath, 2020; Schwartzman, 2022; Zech et al, 2022), an FIS client rated measure (Santos et al, 2023), and FIS for multiple responses that is used to rate therapists as they respond to changing therapeutic situations (Finkelstein, 2022) or high risk situations such as suicidal clients (Duffy, 2022). In addition, the FIS items have been extracted and used for other purposes such as in surveys (e.g., Lin et al, 2021).…”

Evaluating the effectiveness of multimodal psychotherapy training program for medical students in China: study protocol for a randomized controlled trial (Preprint)

“…2) hope and positive expectations, 3) persuasiveness; 4) emotional expression; 5) warmth, acceptance, and understanding; 6) empathy; 7) alliance-bond capacity; and 8) alliance rupture-repair responsiveness. The items are rated on a 5-point Likert scale ranging from 1 (i.e., skill deficit) to 5 (i.e., optimal presence of skill), with high inter-rater agreement and excellent internal consistency [11,12] . Research indicates that FIS is a reliable measure to evaluate performance and psychotherapeutic outcomes in training and assessing therapists, high FIS indicates superior outcomes and better therapeutic processes [8,9] .…”