A Sequential Contrastive Learning Framework for Robust Dysarthric Speech Recognition

Wu, Lidan; Zong, Daoming; Sun, Shiliang; Zhao, Jing

doi:10.1109/icassp39728.2021.9415017

Cited by 8 publications

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Wu et al [83] proposed a contrastive learning framework to capture the acoustic variations of dysarthric speech, aiming to obtain robust recognition results of dysarthric speech. This study also explored data augmentation strategies to alleviate the scarcity of speech data.…”

Section: Acoustic Model Of Asr For Dysarthric Speechmentioning

confidence: 99%

A Survey of Automatic Speech Recognition for Dysarthric Speech

Qian,

Xiao

2023

Electronics

View full text Add to dashboard Cite

Dysarthric speech has several pathological characteristics, such as discontinuous pronunciation, uncontrolled volume, slow speech, explosive pronunciation, improper pauses, excessive nasal sounds, and air-flow noise during pronunciation, which differ from healthy speech. Automatic speech recognition (ASR) can be very helpful for speakers with dysarthria. Our research aims to provide a scoping review of ASR for dysarthric speech, covering papers in this field from 1990 to 2022. Our survey found that the development of research studies about the acoustic features and acoustic models of dysarthric speech is nearly synchronous. During the 2010s, deep learning technologies were widely applied to improve the performance of ASR systems. In the era of deep learning, many advanced methods (such as convolutional neural networks, deep neural networks, and recurrent neural networks) are being applied to design acoustic models and lexical and language models for dysarthric-speech-recognition tasks. Deep learning methods are also used to extract acoustic features from dysarthric speech. Additionally, this scoping review found that speaker-dependent problems seriously limit the generalization applicability of the acoustic model. The scarce available speech data cannot satisfy the amount required to train models using big data.

show abstract

Section: Acoustic Model Of Asr For Dysarthric Speechmentioning

confidence: 99%

A Survey of Automatic Speech Recognition for Dysarthric Speech

Qian,

Xiao

2023

Electronics

View full text Add to dashboard Cite

show abstract

“…Wu, et al [97] proposed a contrastive learning framework to capture the acoustic variations of dysarthric speech, aiming to obtain robust recognition results of dysarthric speech. Their study also explored data augmentation strategies to alleviate the scarcity of speech data.…”

Section: Deep Learning Technologies Of Asr For Dysarthric Speechmentioning

confidence: 99%

A survey of technologies for automatic Dysarthric speech recognition

Qian,

Xiao,

2023

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

Speakers with dysarthria often struggle to accurately pronounce words and effectively communicate with others. Automatic speech recognition (ASR) is a powerful tool for extracting the content from speakers with dysarthria. However, the narrow concept of ASR typically only covers technologies that process acoustic modality signals. In this paper, we broaden the scope of this concept that the generalized concept of ASR for dysarthric speech. Our survey discussed the systems encompassed acoustic modality processing, articulatory movements processing and audio-visual modality fusion processing in the application of recognizing dysarthric speech. Contrary to previous surveys on dysarthric speech recognition, we have conducted a systematic review of the advancements in this field. In particular, we introduced state-of-the-art technologies to supplement the survey of recent research during the era of multi-modality fusion in dysarthric speech recognition. Our survey found that audio-visual fusion technologies perform better than traditional ASR technologies in the task of dysarthric speech recognition. However, training audio-visual fusion models requires more computing resources, and the available data corpus for dysarthric speech is limited. Despite these challenges, state-of-the-art technologies show promising potential for further improving the accuracy of dysarthric speech recognition in the future.

show abstract

“…First of all, ASR performs poorly on impaired speech [4,5]. Though state-ofthe-art ASR models with deep learning approaches have advanced greatly, they are challenged by the data scarcity issue when it comes to dysarthric speech [6]. Recording speech from individuals with dysarthria is hindered by greater recruitment efforts as well as speaker exhaustion.…”

Section: Introductionmentioning

confidence: 99%

“…Recording speech from individuals with dysarthria is hindered by greater recruitment efforts as well as speaker exhaustion. Moreover, vocal characteristics vary greatly between impaired speakers and impairment types [6]. Therefore, the lack of data and the large speech variation lead to a drastic drop in ASR performance.…”

Section: Introductionmentioning

confidence: 99%

Benefits of pre-trained mono- and cross-lingual speech representations for spoken language understanding of Dutch dysarthric speech

Wang

hamme

2023

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

With the rise of deep learning, spoken language understanding (SLU) for command-and-control applications such as a voice-controlled virtual assistant can offer reliable hands-free operation to physically disabled individuals. However, due to data scarcity, it is still a challenge to process dysarthric speech. Pre-training (part of) the SLU model with supervised automatic speech recognition (ASR) targets or with self-supervised learning (SSL) may help to overcome a lack of data, but no research has shown which pre-training strategy performs better for SLU on dysarthric speech and to which extent the SLU task benefits from knowledge transfer from pre-training with dysarthric acoustic tasks. This work aims to compare different mono- or cross-lingual pre-training (supervised and unsupervised) methodologies and quantitatively investigates the benefits of pre-training for SLU tasks on Dutch dysarthric speech. The designed SLU systems consist of a pre-trained speech representations encoder and a SLU decoder to map encoded features to intents. Four types of pre-trained encoders, a mono-lingual time-delay neural network (TDNN) acoustic model, a mono-lingual transformer acoustic model, a cross-lingual transformer acoustic model (Whisper), and a cross-lingual SSL Wav2Vec2.0 model (XLSR-53), are evaluated complemented with three types of SLU decoders: non-negative matrix factorization (NMF), capsule networks, and long short-term memory (LSTM) networks. The acoustic analysis of the four pre-trained encoders are tested on Dutch dysarthric home-automation data with word error rate (WER) results to investigate the correlations of the dysarthric acoustic task (ASR) and the semantic task (SLU). By introducing the intelligibility score (IS) as a metric of the impairment severity, this paper further quantitatively analyzes dysarthria-severity-dependent models for SLU tasks.

show abstract

A Sequential Contrastive Learning Framework for Robust Dysarthric Speech Recognition

Cited by 8 publications

References 19 publications

A Survey of Automatic Speech Recognition for Dysarthric Speech

A Survey of Automatic Speech Recognition for Dysarthric Speech

A survey of technologies for automatic Dysarthric speech recognition

Benefits of pre-trained mono- and cross-lingual speech representations for spoken language understanding of Dutch dysarthric speech

Contact Info

Product

Resources

About