Detection of Lexical Stress Errors in Non-Native (L2) English with Data Augmentation and Attention

Korzekwa, Daniel; Barra-Chicote, Roberto; Zaporowski, Szymon; Beringer, Grzegorz; Lorenzo-Trueba, Jaime; Serafinowicz, Alicja; Droppo, Jasha; Drugman, Thomas; Kostek, Bożena

doi:10.21437/interspeech.2021-86

Cited by 10 publications

(12 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Compared to [22], the proposed mixup is able to generate as many as artificial word samples with a small amount of taskrelated unlabeled data. Different from [16,26], where the label of generated training samples is binary(e.g., correct or mispronounced), proposed mixup can generate continuous scores for augmented data.…”

Section: Mixup Data Generationmentioning

confidence: 99%

“…The comparison in [25] show that the largest nonnative corpus contains 90,841 utterances, but it is not publicly available. To tackle the limited non-native data problem, data augmentation techniques have been investigated in some previous work [16,22,26]. In [26], Text-To-Speech (TTS) was used to synthesize "incorrect stress" samples on top of modified lexical stress.…”

Section: Introductionmentioning

confidence: 99%

“…To tackle the limited non-native data problem, data augmentation techniques have been investigated in some previous work [16,22,26]. In [26], Text-To-Speech (TTS) was used to synthesize "incorrect stress" samples on top of modified lexical stress. In [16], the author created negative training samples by modifying the canonical text as the modified text forms a mismatched pair with its original word-level speech.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Improving Non-native Word-level Pronunciation Scoring with Phone-level Mixup Data Augmentation and Multi-source Information

Fu¹,

Gao²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep learning-based pronunciation scoring models highly rely on the availability of the annotated non-native data, which is costly and has scalability issues. To deal with the data scarcity problem, data augmentation is commonly used for model pretraining. In this paper, we propose a phone-level mixup, a simple yet effective data augmentation method, to improve the performance of word-level pronunciation scoring. Specifically, given a phoneme sequence from lexicon, the artificial augmented word sample can be generated by randomly sampling from the corresponding phone-level features in training data, while the word score is the average of their GOP scores. Benefit from the arbitrary phone-level combination, the mixup is able to generate any word with various pronunciation scores. Moreover, we utilize multi-source information (e.g., MFCC and deep features) to further improve the scoring system performance. The experiments conducted on the Speechocean762 show that the proposed system outperforms the baseline by adding the mixup data for pretraining, with Pearson correlation coefficients (PCC) increasing from 0.567 to 0.61. The results also indicate that proposed method achieves similar performance by using 1/10 unlabeled data of baseline. In addition, the experimental results also demonstrate the efficiency of our proposed multisource approach.

show abstract

Section: Mixup Data Generationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Improving Non-native Word-level Pronunciation Scoring with Phone-level Mixup Data Augmentation and Multi-source Information

Fu¹,

Gao²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The label information also includes an ID representing the pronounced words from 0 to 99. The dataset is motivated by the lack of data for training or finetuning speech representation models, such as the "wav2vec" model [2], and "HuBERT" [3], and to facilitate the development of Arabic language pronunciation mistake identifiers [4], [5], [6].…”

Section: Dataset Descriptionmentioning

confidence: 99%

Dataset_Arabic_speech_mispronunciation_detection

2021

View full text Add to dashboard Cite

“…In recent years, Computer Aided Pronunciation Training (CAPT) tools have been developed to provide diagnosis and feedback on phonetic-level errors (phoneme substitution, deletion, insertion [1,2,3,4,5]) and prosodic-level errors (e.g. lexical stress, intonation [6]). In this study, we focus on detecting phonetic-level pronunciation errors for L2 speech intelligibility and accentedness assessment.…”

Section: Introductionmentioning

confidence: 99%

Improving Mispronunciation Detection with Wav2vec2-based Momentum Pseudo-Labeling for Accentedness and Intelligibility Assessment

Yang¹,

Hirschi²,

Looney³

et al. 2022

Preprint

View full text Add to dashboard Cite

Current leading mispronunciation detection and diagnosis (MDD) systems achieve promising performance via end-to-end phoneme recognition. One challenge of such end-to-end solutions is the scarcity of human-annotated phonemes on natural L2 speech. In this work, we leverage unlabeled L2 speech via a pseudo-labeling (PL) procedure and extend the fine-tuning approach based on pre-trained self-supervised learning (SSL) models. Specifically, we use Wav2vec 2.0 as our SSL model, and fine-tune it using original labeled L2 speech samples plus the created pseudo-labeled L2 speech samples. Our pseudo labels are dynamic and are produced by an ensemble of the online model on-the-fly, which ensures that our model is robust to pseudo label noise. We show that fine-tuning with pseudo labels gains a 5.35% phoneme error rate reduction and 2.48% MDD F1 score improvement over a labeled-samples-only fine-tuning baseline. The proposed PL method is also shown to outperform conventional offline PL methods. Compared to the state-of-theart MDD systems, our MDD solution achieves a more accurate and consistent phonetic error diagnosis. In addition, we conduct an open test on a separate UTD-4Accents dataset, where our system recognition outputs show a strong correlation with human perception, based on accentedness and intelligibility.

show abstract

Detection of Lexical Stress Errors in Non-Native (L2) English with Data Augmentation and Attention

Cited by 10 publications

References 0 publications

Improving Non-native Word-level Pronunciation Scoring with Phone-level Mixup Data Augmentation and Multi-source Information

Improving Non-native Word-level Pronunciation Scoring with Phone-level Mixup Data Augmentation and Multi-source Information

Dataset_Arabic_speech_mispronunciation_detection

Improving Mispronunciation Detection with Wav2vec2-based Momentum Pseudo-Labeling for Accentedness and Intelligibility Assessment

Contact Info

Product

Resources

About