Multimodal Deep Learning Models for Detecting Dementia From Speech and Transcripts

Abstract

Alzheimer's dementia (AD) entails negative psychological, social, and economic consequences not only for the patients but also for their families, relatives, and society in general. Despite the significance of this phenomenon and the importance for an early diagnosis, there are still limitations. Specifically, the main limitation is pertinent to the way the modalities of speech and transcripts are combined in a single neural network. Existing research works add/concatenate the image and text representations, employ majority voting approaches or average the predictions after training many textual and speech models separately. To address these limitations, in this article we present some new methods to detect AD patients and predict the Mini-Mental State Examination (MMSE) scores in an end-to-end trainable manner consisting of a combination of BERT, Vision Transformer, Co-Attention, Multimodal Shifting Gate, and a variant of the self-attention mechanism. Specifically, we convert audio to Log-Mel spectrograms, their delta, and delta-delta (acceleration values). First, we pass each transcript and image through a BERT model and Vision Transformer, respectively, adding a co-attention layer at the top, which generates image and word attention simultaneously. Secondly, we propose an architecture, which integrates multimodal information to a BERT model via a Multimodal Shifting Gate. Finally, we introduce an approach to capture both the inter- and intra-modal interactions by concatenating the textual and visual representations and utilizing a self-attention mechanism, which includes a gate model. Experiments conducted on the ADReSS Challenge dataset indicate that our introduced models demonstrate valuable advantages over existing research initiatives achieving competitive results in both the AD classification and MMSE regression task. Specifically, our best performing model attains an accuracy of 90.00% and a Root Mean Squared Error (RMSE) of 3.61 in the AD classification task and MMSE regression task, respectively, achieving a new state-of-the-art performance in the MMSE regression task.

Neural Computation-Based Methods for the Early Diagnosis and Prognosis of Alzheimer’s Disease Not Using Neuroimaging Biomarkers: A Systematic Review

“…Ilias et al [ 149 ] combined a BERT with a ViT and Co-attention to successfully fulfill an AD-nonAD classification task.…”

A remote speech‐based AI system to screen for early Alzheimer's disease via smartphones

“… 33 Prior research indicates that approaches using acoustic speech input are also sensitive to dementia, albeit usually to a lesser extent than approaches using textual speech input. 43 Combining these different information modalities in speech could help to further augment classification sensitivity, 11 , 44 although this improvement is not always shown in practice. 45 Potential directions for future research in the AMYPRED sample include linguistic, temporal, and acoustic aspects of speech.…”

“…However, speech is highly multidimensional, incorporating not only linguistic but also acoustic and temporal features, which could confer additional sensitivity to cognitive impairment and amyloid positivity 33 . Prior research indicates that approaches using acoustic speech input are also sensitive to dementia, albeit usually to a lesser extent than approaches using textual speech input 43 . Combining these different information modalities in speech could help to further augment classification sensitivity, 11,44 although this improvement is not always shown in practice 45 .…”

Deep learning applications in vascular dementia using neuroimaging

“…They also performed occlusion analysis to reveal the most significant brain areas for the dementia subtypes classification task. Additionally, researchers also used cognitive data [56], speech features [57,58], and fundus photographs data [59] in deep learning models for dementia diagnosis.…”