Improving Classification of Infrequent Cognitive Distortions: Domain-Specific Model vs. Data Augmentation

Abstract

Cognitive distortions are counterproductive patterns of thinking that are one of the targets of cognitive behavioral therapy (CBT). These can be challenging for clinicians to detect, especially those without extensive CBT training or supervision. Text classification methods can approximate expert clinician judgment in the detection of frequently occurring cognitive distortions in text-based therapy messages. However, performance with infrequent distortions is relatively poor. In this study, we address this sparsity problem with two approaches: Data Augmentation and Domain-Specific Model. The first approach includes Easy Data Augmentation, back translation, and mixup techniques. The second approach utilizes a domain-specific pretrained language model, MentalBERT. To examine the viability of different data augmentation methods, we utilized a real-world dataset of texts between therapists and clients diagnosed with serious mental illness that was annotated for distorted thinking. We found that with optimized parameter settings, mixup was helpful for rare classes. Performance improvements with an augmented model, MentalBERT, exceed those obtained with data augmentation.

Natural language processing for mental health interventions: a systematic review and research framework

“…Although requiring massive text corpora to initially train on masked language, language models build linguistic representations that can then be fine-tuned to downstream clinical tasks [ 69 ]. Applications examined include fine-tuning BERT for domain adaptation to mental health language (MentalBERT) [ 70 ], for sentiment analysis via transfer learning (e.g., using the GoEmotions corpus) [ 71 ], and detection of topics [ 72 ]. Generative language models were used for revising interventions [ 73 ], session summarizations [ 74 ], or data augmentation for model training [ 70 ].…”

“…Applications examined include fine-tuning BERT for domain adaptation to mental health language (MentalBERT) [ 70 ], for sentiment analysis via transfer learning (e.g., using the GoEmotions corpus) [ 71 ], and detection of topics [ 72 ]. Generative language models were used for revising interventions [ 73 ], session summarizations [ 74 ], or data augmentation for model training [ 70 ].…”

“…Eleven SMI applications used NLP markers to identify psychosis [ 51 , 81 , 84 , 85 ] and bipolar [ 86 ] diagnoses (Accuracy 0.70 and 0.85), monitor symptoms [ 75 , 87 ], detect psychotic episodes ( F 1 = 0.99) [ 88 ] and psychosis onset (AUC = 0.72) [ 89 ]. Negative symptoms [ 87 , 90 ] and cognitive distortions [ 70 ] for SMI were detected using linguistic features, including connectedness emerging from graph analytics [ 90 , 91 ]. Associations between linguistic features and neuroimaging were also examined [ 91 , 92 ].…”

“…The most reliable route to achieving statistical power and representativeness is more data, which is challenging in healthcare given regulations for data confidentiality and ethical considerations of patient privacy. Technical solutions to leverage low resource clinical datasets include augmentation [ 70 ], out-of-domain pre-training [ 68 , 70 ], and meta-learning [ 119 , 143 ]. However, findings from our review suggest that these methods do not necessarily improve performance in clinical domains [ 68 , 70 ] and, thus, do not substitute the need for large corpora.…”

“…Technical solutions to leverage low resource clinical datasets include augmentation [ 70 ], out-of-domain pre-training [ 68 , 70 ], and meta-learning [ 119 , 143 ]. However, findings from our review suggest that these methods do not necessarily improve performance in clinical domains [ 68 , 70 ] and, thus, do not substitute the need for large corpora. As noted, data from large service providers are critical for continued NLP progress, but privacy concerns require additional oversight and planning.…”