Automated evaluation of psychotherapy skills using speech and language technologies

Abstract

With the growing prevalence of psychological interventions, it is vital to have measures which rate the effectiveness of psychological care to assist in training, supervision, and quality assurance of services. Traditionally, quality assessment is addressed by human raters who evaluate recorded sessions along specific dimensions, often codified through constructs relevant to the approach and domain. This is, however, a cost-prohibitive and time-consuming method that leads to poor feasibility and limited use in real-world settings. To facilitate this process, we have developed an automated competency rating tool able to process the raw recorded audio of a session, analyzing who spoke when, what they said, and how the health professional used language to provide therapy. Focusing on a use case of a specific type of psychotherapy called "motivational interviewing", our system gives comprehensive feedback to the therapist, including information about the dynamics of the session (e.g., therapist's vs. client's talking time), low-level psychological language descriptors (e.g., type of questions asked), as well as other high-level behavioral constructs (e.g., the extent to which the therapist understands the clients' perspective). We describe our platform and its performance using a dataset of more than 5000 recordings drawn from its deployment in a real-world clinical setting used to assist training of new therapists. Widespread use of automated psychotherapy rating tools may augment experts' capabilities by providing an avenue for more effective training and skill improvement, eventually leading to more positive clinical outcomes.

Machine learning and natural language processing in psychotherapy research: Alliance as example use case.

“…The modules of the speech pipeline have been adapted with the Kaldi speech recognition toolkit (Povey et al, 2011) using psychotherapy sessions provided by the same counseling center, but not used for the alliance prediction, thus not inducing bias. A similar system architecture is described in Xiao et al (2016) and Flemotomos et al (2019).…”

Multimodal Clustering with Role Induced Constraints for Speaker Diarization

“…Psychotherapy sessions. We use a collection of psychotherapy sessions recorded at a university counseling center (UCC), and specifically the sessions in the sets denoted as UCCtrain, UCC dev and UCCtest 1 in [24]. Each session is a dyadic conversation between a therapist and a patient, thus falls under case (3) according to the categorization of Section 3.…”

Uncovering the linguistic characteristics of psychotherapy: a computational approach to measure therapist language timing, responsiveness, and consistency

“…Studies of the psychotherapy process attempt to understand what happens during the sessions that may explain patient improvement 8 . The chief method used since the 1950s to evaluate therapist behavior in therapy sessions is to have trained humans identify clinically meaningful therapist utterances in transcripts, and draw conclusions based on observed patterns [9][10][11][12] . Although useful, relying solely on human inspection of transcripts is not likely to meet demands for improved reproducibility and scalability in psychotherapy process research 11,[13][14][15][16][17] .…”

“…Computational approaches using natural language processing offer the potential to move past human limits of attention and reproducibility 11,[18][19][20][21][22] . Improvements in computational power, the growing ease of recording and transcribing therapy sessions, and a shift to computer-mediated communication in healthcare (i.e., telehealth) make this feasible 11,14,23,24 . Early work is promising, but does not yet translate to best practices for improved patient outcomes or provide a clear direction for therapist training 11,19,[25][26][27][28] .…”

“…Improvements in computational power, the growing ease of recording and transcribing therapy sessions, and a shift to computer-mediated communication in healthcare (i.e., telehealth) make this feasible 11,14,23,24 . Early work is promising, but does not yet translate to best practices for improved patient outcomes or provide a clear direction for therapist training 11,19,[25][26][27][28] . Methodological improvements are needed to bridge divisions between theoretical schools of thought (e.g., Cognitive Behavioral, Interpersonal, Psychodynamic, Counseling) as to which therapist language patterns correlate with favorable therapy outcomes 4,6,13,15,20,24,29,30 .…”