A Study into Pre-Training Strategies for Spoken Language Understanding on Dysarthric Speech

Wang, Pu; BabaAli, Bagher; hamme, Hugo Van

doi:10.21437/interspeech.2021-1720

Cited by 8 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Inspired by these, we explore pre-training of a TDNN acoustic model on a publicly available dysarthric speech corpus with ASR targets and then extract layer activations of the well-trained TDNN model as BNFs for dysarthric end-user utterances. The TDNN-based acoustic model is evaluated with capsule networks in [36] to predict SLU intent labels. To verify that we are learning something from the TDNN-based acoustic model other than just the idiosyncrasies of a particular decoding model, we will extensively evaluate it with three types of SLU decoders for semantic inference: the NMF model, the multilayer capsule network model, and the LSTM model.…”

Section: Contributionsmentioning

confidence: 99%

Benefits of pre-trained mono- and cross-lingual speech representations for spoken language understanding of Dutch dysarthric speech

Wang

hamme

2023

J AUDIO SPEECH MUSIC PROC.

Self Cite

View full text Add to dashboard Cite

With the rise of deep learning, spoken language understanding (SLU) for command-and-control applications such as a voice-controlled virtual assistant can offer reliable hands-free operation to physically disabled individuals. However, due to data scarcity, it is still a challenge to process dysarthric speech. Pre-training (part of) the SLU model with supervised automatic speech recognition (ASR) targets or with self-supervised learning (SSL) may help to overcome a lack of data, but no research has shown which pre-training strategy performs better for SLU on dysarthric speech and to which extent the SLU task benefits from knowledge transfer from pre-training with dysarthric acoustic tasks. This work aims to compare different mono- or cross-lingual pre-training (supervised and unsupervised) methodologies and quantitatively investigates the benefits of pre-training for SLU tasks on Dutch dysarthric speech. The designed SLU systems consist of a pre-trained speech representations encoder and a SLU decoder to map encoded features to intents. Four types of pre-trained encoders, a mono-lingual time-delay neural network (TDNN) acoustic model, a mono-lingual transformer acoustic model, a cross-lingual transformer acoustic model (Whisper), and a cross-lingual SSL Wav2Vec2.0 model (XLSR-53), are evaluated complemented with three types of SLU decoders: non-negative matrix factorization (NMF), capsule networks, and long short-term memory (LSTM) networks. The acoustic analysis of the four pre-trained encoders are tested on Dutch dysarthric home-automation data with word error rate (WER) results to investigate the correlations of the dysarthric acoustic task (ASR) and the semantic task (SLU). By introducing the intelligibility score (IS) as a metric of the impairment severity, this paper further quantitatively analyzes dysarthria-severity-dependent models for SLU tasks.

show abstract

Section: Contributionsmentioning

confidence: 99%

Benefits of pre-trained mono- and cross-lingual speech representations for spoken language understanding of Dutch dysarthric speech

Wang

hamme

2023

J AUDIO SPEECH MUSIC PROC.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The mode of meaning included the specific speech inputs into the proposed models. Studies that solely used speech features [15,[24][25][26][27][28][29][30][31][32][33][34][35] were heavily speaker-dependent on their approach and tended to lean more toward intelligibility of the dysarthric speaker than their comprehensibility.…”

Section: Mode Of Meaning Extraction Usedmentioning

confidence: 99%

“…Finally, studies that used a hybrid approach [23,33,34,45,48,49] combined MFCC with variations of vector encoding. These studies were characterized by variations of models, such as adversarial networks, support vector machines, gated recurrent unit and convolutional neural networks, and hidden Markov models.…”

Section: Nature Of Speech Representations Usedmentioning

confidence: 99%

Models and Approaches for Comprehension of Dysarthric Speech Using Natural Language Processing: Systematic Review

Alaka¹,

Kasamani²

2023

JMIR Rehabil Assist Technol

View full text Add to dashboard Cite

Background Speech intelligibility and speech comprehension for dysarthric speech has attracted much attention recently. Dysarthria is characterized by irregularities in the speed, strength, pitch, breath control, range, steadiness, and accuracy of muscle movements required for articulatory aspects of speech production. Objective This study examined the contributions made by other studies involved in dysarthric speech comprehension. We focused on the modes of meaning extraction used in generalizing speaker-listener underpinnings in light of semantic ontology extraction as a desired technique, applied method types, speech representations used, and databases sourced from. Methods This study involved a systematic literature review using 7 electronic databases: Cochrane Database of Systematic Reviews, Web of Science Core Collection, Scopus, PubMed, ACM, IEEE Xplore, and Google Scholar. The main eligibility criterion was the extraction of meaning from dysarthric speech using natural language processing or understanding approaches to improve on dysarthric speech comprehension. In total, out of 834 search results, 30 studies that matched the eligibility requirements were acquired following screening by 2 independent reviewers, with a lack of consensus being resolved through joint discussion or consultation with a third party. In order to evaluate the studies’ methodological quality, the risk of bias assessment was based on the Cochrane risk-of-bias tool version 2 (RoB2) with 23 of the studies (77%) registering low risk of bias and 7 studies (33%) raising some concern over the risk of bias. The overall quality assessment of the study was done using TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis). Results Following a review of 30 primary studies, this study revealed that the reviewed studies focused on natural language understanding or clinical approaches, with an increase in proposed solutions from 2020 onwards. Most studies relied on speaker-dependent speech features, while others used speech patterns, semantic knowledge, or hybrid approaches. The prevalent use of vector representation aligned with natural language understanding models, while Mel-frequency cepstral coefficient representation and no representation approaches were applied in neural networks. Hybrid representation studies aimed to reconstruct dysarthric speech or improve comprehension. Comprehensive databases, like TORGO and UA-Speech, were commonly used in combination with other curated databases, while primary data was preferred for specific or unique research objectives. Conclusions We found significant gaps in dysarthric speech comprehension characterized by the lack of inclusion of important listener or speech-independent features in the speech representations, mode of extraction, and data sources used. Further research is therefore proposed regarding the formulation of models that accommodate listener and speech-independent features through semantic ontologies that will be useful in the inclusion of key features of listener and speech-independent features for meaning extraction of dysarthric speech.

show abstract

“…To adapt to the Dutch Domotica data, we pre-train this model accompanied with CTC loss using Dutch Copas disordered speech [30]. Kaldi is a TDNN-F-based model proposed in [31], pre-trained on the Dutch Copas data as well. All compared models are combined with the same capsule network decoder for intent classification as described in the Section.2.4.…”

Section: Domotica Datasetmentioning

confidence: 99%

“…All compared models are combined with the same capsule network decoder for intent classification as described in the Section.2.4. [7] 0.931 27M + 0.7M Kaldi (pre-training) [31] 0.939 19M + 0.7M…”

Section: Domotica Datasetmentioning

confidence: 99%

Bottleneck Low-rank Transformers for Low-resource Spoken Language Understanding

Wang¹,

hamme²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

End-to-end spoken language understanding (SLU) systems benefit from pretraining on large corpora, followed by fine-tuning on application-specific data. The resulting models are too large for on-edge applications. For instance, BERT-based systems contain over 110M parameters. Observing the model is overparameterized, we propose lean transformer structure where the dimension of the attention mechanism is automatically reduced using group sparsity. We propose a variant where the learned attention subspace is transferred to an attention bottleneck layer. In a low-resource setting and without pre-training, the resulting compact SLU model achieves accuracies competitive with pre-trained large models.

show abstract

A Study into Pre-Training Strategies for Spoken Language Understanding on Dysarthric Speech

Cited by 8 publications

References 12 publications

Benefits of pre-trained mono- and cross-lingual speech representations for spoken language understanding of Dutch dysarthric speech

Benefits of pre-trained mono- and cross-lingual speech representations for spoken language understanding of Dutch dysarthric speech

Models and Approaches for Comprehension of Dysarthric Speech Using Natural Language Processing: Systematic Review

Bottleneck Low-rank Transformers for Low-resource Spoken Language Understanding

Contact Info

Product

Resources

About