Pause-Encoded Language Models for Recognition of Alzheimer’s Disease and Emotion

Yuan, Jiahong; Cai, Xingyu; Church, Kenneth

doi:10.1109/icassp39728.2021.9413548

Cited by 4 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…WA ↑ UA ↑ MHA-3 [46] 74.0 75.3 F-III [47] 79.2 80.5 AugPauseRoBERTa [48] 72. This is because BERT encodes different properties for each layer.…”

Section: Methodsmentioning

confidence: 99%

CMSBERT-CLR: Context-driven Modality Shifting BERT with Contrastive Learning for linguistic, visual, acoustic Representations

Kim

2022

2022 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

Multimodal sentiment analysis has become an increasingly popular research area as the demand for multimodal online content is growing. For multimodal sentiment analysis, words can have different meanings depending on the linguistic context and non-verbal information, so it is crucial to understand the meaning of the words accordingly. In addition, the word meanings should be interpreted within the whole utterance context that includes nonverbal information. In this paper, we present a Context-driven Modality Shifting BERT with Contrastive Learning for linguistic, visual, acoustic Representations (CMSBERT-CLR), which incorporates the whole context's non-verbal and verbal information and aligns modalities more effectively through contrastive learning. First, we introduce a Context-driven Modality Shifting (CMS) to incorporate the non-verbal and verbal information within the whole context of the sentence utterance. Then, for improving the alignment of different modalities within a common embedding space, we apply contrastive learning. Furthermore, we use an exponential moving average parameter and label smoothing as optimization strategies, which can make the convergence of the network more stable and increase the flexibility of the alignment. In our experiments, we demonstrate that our approach achieves stateof-the-art results.

show abstract

“…WA ↑ UA ↑ MHA-3 [46] 74.0 75.3 F-III [47] 79.2 80.5 AugPauseRoBERTa [48] 72. This is because BERT encodes different properties for each layer.…”

Section: Methodsmentioning

confidence: 99%

CMSBERT-CLR: Context-driven Modality Shifting BERT with Contrastive Learning for linguistic, visual, acoustic Representations

Kim

2022

2022 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

show abstract

“…16 Other reports based on language and speech features ranged from 70% to 89.6% accuracy. [17][18][19][20][21][22][23][24][25] While previous studies have shown promising results with speech features classifying the amnestic AD presentation versus HC, a clinically meaningful challenge that has not been addressed is distinguishing underlying ADNC from FTLD pathology in individuals presenting with non-amnestic phenotypes. In this study, we used digital speech features that were extracted with automatic lexical and acoustic pro- with different feature sets and applied an interpretable artificial intelligence (AI) approach to answer these questions.…”

Section: Introductionmentioning

confidence: 99%

“…A more recent study used a large dataset and a natural language processing (NLP) approach to differentiate individuals with dementia from HC with 87.1% accuracy and to distinguish individuals with mild cognitive impairment (MCI) from HC with 71.2% accuracy 16 . Other reports based on language and speech features ranged from 70% to 89.6% accuracy 17–25 …”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Automatic classification of AD pathology in FTD phenotypes using natural speech

Cho,

Olm,

Ash

et al. 2024

Alzheimer's & Dementia

View full text Add to dashboard Cite

INTRODUCTIONScreening for Alzheimer's disease neuropathologic change (ADNC) in individuals with atypical presentations is challenging but essential for clinical management. We trained automatic speech‐based classifiers to distinguish frontotemporal dementia (FTD) patients with ADNC from those with frontotemporal lobar degeneration (FTLD).METHODSWe trained automatic classifiers with 99 speech features from 1 minute speech samples of 179 participants (ADNC = 36, FTLD = 60, healthy controls [HC] = 89). Patients’ pathology was assigned based on autopsy or cerebrospinal fluid analytes. Structural network‐based magnetic resonance imaging analyses identified anatomical correlates of distinct speech features.RESULTSOur classifier showed 0.88 0.03 area under the curve (AUC) for ADNC versus FTLD and 0.93 0.04 AUC for patients versus HC. Noun frequency and pause rate correlated with gray matter volume loss in the limbic and salience networks, respectively.DISCUSSIONBrief naturalistic speech samples can be used for screening FTD patients for underlying ADNC in vivo. This work supports the future development of digital assessment tools for FTD.Highlights We trained machine learning classifiers for frontotemporal dementia patients using natural speech. We grouped participants by neuropathological diagnosis (autopsy) or cerebrospinal fluid biomarkers. Classifiers well distinguished underlying pathology (Alzheimer's disease vs. frontotemporal lobar degeneration) in patients. We identified important features through an explainable artificial intelligence approach. This work lays the groundwork for a speech‐based neuropathology screening tool.

show abstract

End-to-End ASR-Enhanced Neural Network for Alzheimer’s Disease Diagnosis

Gui

Chen

et al. 2022

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Pause-Encoded Language Models for Recognition of Alzheimer’s Disease and Emotion

Cited by 4 publications

References 25 publications

CMSBERT-CLR: Context-driven Modality Shifting BERT with Contrastive Learning for linguistic, visual, acoustic Representations

CMSBERT-CLR: Context-driven Modality Shifting BERT with Contrastive Learning for linguistic, visual, acoustic Representations

Automatic classification of AD pathology in FTD phenotypes using natural speech

End-to-End ASR-Enhanced Neural Network for Alzheimer’s Disease Diagnosis

Contact Info

Product

Resources

About