XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale

Babu, Arun; Wang, Changhan; Tjandra, Andros; Lakhotia, Kushal; Xu, Qiantong; Goyal, Naman; Singh, Kritika; Platen, Patrick von; Saraf, Yatharth; Pino, Juan; Baevski, Alexei; Conneau, Alexis; Auli, Michael

doi:10.48550/arxiv.2111.09296

Cited by 51 publications

(121 citation statements)

References 30 publications

Supporting

Mentioning

118

Contrasting

Order By: Relevance

“…We use XLS-R -U as pretraining data and finetune it on MLS-10hrs. As shown in Table 3, our baseline w2v-BERT already outperform previous strong model from XLS-R(2B) (Babu et al, 2021). The average WER further bring down by 3% relative by using the proposed BEST-RQ.…”

Section: Results On Mls-10hrsmentioning

confidence: 72%

“…XLS-R unsupervised data (XLS-R -U) Our public unlabeled speech data follows the pre-training data used for XLS-R (Babu et al, 2021) with one major difference: we do not use any data from VoxLingua-107 due to license constraint. In total, we utilize approximately 429k hours of unlabeled speech data in 51 1 languages.…”

Section: Datamentioning

confidence: 99%

“…XLSR-53 (Conneau et al, 2020) 14.6 8.4 12.8 12.5 8.9 13.4 18.2 21.2 13.8 XLS-R(0.3B) (Babu et al, 2021) 15.9 9.0 13.5 12.4 8.1 13.1 17.0 13.9 12.8 XLS-R(1B) (Babu et al, 2021) 12.9 7.4 11.6 10.2 7.1 12.0 15.8 10.5 10.9 XLS-R(2B) (Babu et al, 2021) 14.0 7.6 11.8 10.0 6.9 12.1 15.6 9.8 11.0 w2v-BERT (0.6B) 12.7 7.0 12.6 8.9 5.9 10.3 14.6 6.9 9.9 BEST-RQ (Ours, 0.6B) 12.8 7.4 12.7 9.6 5.4 9.9 12.1 7.1 9.6…”

Section: Mls-10hrsmentioning

confidence: 99%

See 2 more Smart Citations

Self-supervised Learning with Random-projection Quantizer for Speech Recognition

Chiu¹,

Qin²,

Yu³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present a simple and effective self-supervised learning approach for speech recognition. The approach learns a model to predict the masked speech signals, in the form of discrete labels generated with a random-projection quantizer. In particular the quantizer projects speech inputs with a randomly initialized matrix, and does a nearest-neighbor lookup in a randomly-initialized codebook. Neither the matrix nor the codebook is updated during self-supervised learning. Since the random-projection quantizer is not trained and is separated from the speech recognition model, the design makes the approach flexible and is compatible with universal speech recognition architecture. On LibriSpeech our approach achieves similar word-error-rates as previous work using selfsupervised learning with non-streaming models, and provides lower word-error-rates and latency than wav2vec 2.0 and w2v-BERT with streaming models. On multilingual tasks the approach also provides significant improvement over wav2vec 2.0 and w2v-BERT.

show abstract

Section: Results On Mls-10hrsmentioning

confidence: 72%

Section: Datamentioning

confidence: 99%

Section: Mls-10hrsmentioning

confidence: 99%

See 1 more Smart Citation

Self-supervised Learning with Random-projection Quantizer for Speech Recognition

Chiu¹,

Qin²,

Yu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…On the other side, the first layers are better, but still not optimal, probably due to the low contextual information. We find that middle layers (11)(12)(13)(14)(15)(16)(17)(18)(19)(20) have the most informative representation that can be used by the classifier. More specifically, the output of the 14th layer, achieves the best segmentation, retaining almost 95% of the manual BLEU score.…”

Section: A Appendixmentioning

confidence: 99%

“…After self-supervision, it can be fine-tuned to downstream tasks like ASR. Its multilingual version, XLS-R [13] has been pre-trained on 128 languages using 436k hours of speech data. Audio Segmentation methods.…”

Section: Introductionmentioning

confidence: 99%

SHAS: Approaching optimal Segmentation for End-to-End Speech Translation

Tsiamas¹,

Gállego²,

Fonollosa³

et al. 2022

Preprint

View full text Add to dashboard Cite

Speech translation models are unable to directly process long audios, like TED talks, which have to be split into shorter segments. Speech translation datasets provide manual segmentations of the audios, which are not available in real-world scenarios, and existing segmentation methods usually significantly reduce translation quality at inference time. To bridge the gap between the manual segmentation of training and the automatic one at inference, we propose Supervised Hybrid Audio Segmentation (SHAS), a method that can effectively learn the optimal segmentation from any manually segmented speech corpus. First, we train a classifier to identify the included frames in a segmentation, using speech representations from a pre-trained wav2vec 2.0. The optimal splitting points are then found by a probabilistic Divide-and-Conquer algorithm that progressively splits at the frame of lowest probability until all segments are below a pre-specified length. Experiments on MuST-C and mT-EDx show that the translation of the segments produced by our method approaches the quality of the manual segmentation on 5 languages pairs. Namely, SHAS retains 95-98% of the manual segmentation's BLEU score, compared to the 87-93% of the best existing methods. Our method is additionally generalizable to different domains and achieves high zero-shot performance in unseen languages.

show abstract

Speech Emotion Recognition from Social Media Voice Messages Recorded in the Wild

Gómez-Zaragozá

Marín‐Morales

Parra

et al. 2020

Communications in Computer and Information Science

View full text Add to dashboard Cite

Speech is the most natural way for human communication, carrying the emotional state of the speaker that plays an important role in social interaction. Currently, many instant messaging apps offer the possibility of exchanging voice audios with other users. As a result, a great amount of voice data is generated every day, representing a new challenging approach for speech emotion recognition in real environments. In this study, we investigated emotion recognition from voice messages recorded in the wild using machine-learning algorithms. Unlike most research in this field, which use databases based on emotions evoked in lab environments, simulated by actors or subjectively selected from radio or TV talks, we created an ecological speech dataset with audios from real WhatsApp conversations of 30 Spanish speakers. Four external evaluators labelled each audio in terms of arousal and valence using the Self-Assessment Manikin (SAM) procedure. Pre-processing techniques were applied to the audios and different time and frequency domain features were extracted. Supervised machine learning classifiers were computed using feature reduction and hyper-parameter tuning in order to recognize the affective state of each voice message. The best recognition rate was obtained with Support Vector Machines, achieving 71.37% along the arousal dimension and 70.73% along the valence dimension. These results support the use of emotion recognition models on daily communication apps, helping to understand social human behavior and their interactions with devices in the real world.

show abstract

XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale

Cited by 51 publications

References 30 publications

Self-supervised Learning with Random-projection Quantizer for Speech Recognition

Self-supervised Learning with Random-projection Quantizer for Speech Recognition

SHAS: Approaching optimal Segmentation for End-to-End Speech Translation

Speech Emotion Recognition from Social Media Voice Messages Recorded in the Wild

Contact Info

Product

Resources

About