Explore wav2vec 2.0 for Mispronunciation Detection

Xu, Xiaoshuo; Kang, Yueteng; Cao, Songjun; Lin, Baiquan; Ma, Long

doi:10.21437/interspeech.2021-777

Cited by 34 publications

(14 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Examples include contrastive predictive coding (CPC) [16,29], auto-regressive predictive coding [30], wav2vec [31], HuBERT [32,33], wav2vec 2.0 [12,34] and Wavlm [35], with all showing promising results for a variety of different speech processing tasks. Two particularly popular approaches, HuBERT and wav2vec 2.0, have been applied to automatic speech recognition [12,13], mispronunciation detection [36,37], speaker recognition [38,39] and emotion recognition [40]. The same techniques have been explored in the context of spoofing detection [20,21].…”

Section: Related Workmentioning

confidence: 99%

Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation

Tak,

Todisco,

Wang

et al. 2022

Preprint

View full text Add to dashboard Cite

The performance of spoofing countermeasure systems depends fundamentally upon the use of sufficiently representative training data. With this usually being limited, current solutions typically lack generalisation to attacks encountered in the wild. Strategies to improve reliability in the face of uncontrolled, unpredictable attacks are hence needed. We report in this paper our efforts to use self-supervised learning in the form of a wav2vec 2.0 front-end with fine tuning. Despite initial base representations being learned using only bona fide data and no spoofed data, we obtain the lowest equal error rates reported in the literature for both the ASVspoof 2021 Logical Access and Deepfake databases. When combined with data augmentation, these results correspond to an improvement of almost 90% relative to our baseline system.

show abstract

Section: Related Workmentioning

confidence: 99%

Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation

Tak,

Todisco,

Wang

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Goodness-of-pronunciation (GOP) is among the first DNN-based methods to MDD, which relies on phone posterior outputs from an automatic speech recognizer (ASR) [1,2,23] to evaluate phonetic errors. More recently, end-to-end phoneme recognition has been studied [5,3,4,8,24], among which [4] and [24] also explored fine-tuning Wav2vec 2.0. Our proposed method differs with them in that we investigate the usage of unlabeled target domain speech to enhance MDD performance.…”

Section: Related Workmentioning

confidence: 99%

Improving Mispronunciation Detection with Wav2vec2-based Momentum Pseudo-Labeling for Accentedness and Intelligibility Assessment

Yang¹,

Hirschi²,

Looney³

et al. 2022

Preprint

View full text Add to dashboard Cite

Current leading mispronunciation detection and diagnosis (MDD) systems achieve promising performance via end-to-end phoneme recognition. One challenge of such end-to-end solutions is the scarcity of human-annotated phonemes on natural L2 speech. In this work, we leverage unlabeled L2 speech via a pseudo-labeling (PL) procedure and extend the fine-tuning approach based on pre-trained self-supervised learning (SSL) models. Specifically, we use Wav2vec 2.0 as our SSL model, and fine-tune it using original labeled L2 speech samples plus the created pseudo-labeled L2 speech samples. Our pseudo labels are dynamic and are produced by an ensemble of the online model on-the-fly, which ensures that our model is robust to pseudo label noise. We show that fine-tuning with pseudo labels gains a 5.35% phoneme error rate reduction and 2.48% MDD F1 score improvement over a labeled-samples-only fine-tuning baseline. The proposed PL method is also shown to outperform conventional offline PL methods. Compared to the state-of-theart MDD systems, our MDD solution achieves a more accurate and consistent phonetic error diagnosis. In addition, we conduct an open test on a separate UTD-4Accents dataset, where our system recognition outputs show a strong correlation with human perception, based on accentedness and intelligibility.

show abstract

“…The network learns speech representations from raw audio which can be used in downstream tasks such as speech recognition. Speech representation learning models such as Wav2Vec2.0 [2], and Hubert [3], have shown that learned representations produce state-of-the-art results on a variety of speech tasks: Speaker and language identification [4], emotion recognition [5], spoofing speech detection [6], and second language mispronunciation detection [7]. However, there is a lack of studies regarding the benefits of Wav2Vec or other speech representation models for impaired speech tasks like speech recognition or diagnosis.…”

Section: Introductionmentioning

confidence: 99%

Cross-lingual Self-Supervised Speech Representations for Improved Dysarthric Speech Recognition

Hernandez¹,

Pérez-Toro²,

Nöth³

et al. 2022

Preprint

View full text Add to dashboard Cite

State-of-the-art automatic speech recognition (ASR) systems perform well on healthy speech. However, the performance on impaired speech still remains an issue. The current study explores the usefulness of using Wav2Vec self-supervised speech representations as features for training an ASR system for dysarthric speech. Dysarthric speech recognition is particularly difficult as several aspects of speech such as articulation, prosody and phonation can be impaired. Specifically, we train an acoustic model with features extracted from Wav2Vec, Hubert, and the cross-lingual XLSR model. Results suggest that speech representations pretrained on large unlabelled data can improve word error rate (WER) performance. In particular, features from the multilingual model led to lower WERs than filterbanks (Fbank) or models trained on a single language. Improvements were observed in English speakers with cerebral palsy caused dysarthria (UASpeech corpus), Spanish speakers with Parkinsonian dysarthria (PC-GITA corpus) and Italian speakers with paralysis-based dysarthria (EasyCall corpus). Compared to using Fbank features, XLSR-based features reduced WERs by 6.8%, 22.0%, and 7.0% for the UASpeech, PC-GITA, and EasyCall corpus, respectively.

show abstract

Explore wav2vec 2.0 for Mispronunciation Detection

Cited by 34 publications

References 8 publications

Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation

Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation

Improving Mispronunciation Detection with Wav2vec2-based Momentum Pseudo-Labeling for Accentedness and Intelligibility Assessment

Cross-lingual Self-Supervised Speech Representations for Improved Dysarthric Speech Recognition

Contact Info

Product

Resources

About