XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale

Babu, Arun; Wang, Changhan; Tjandra, Andros; Lakhotia, Kushal; Xu, Qiantong; Goyal, Naman; Singh, Kritika; Platen, Patrick von; Saraf, Yatharth; Pino, Juan; Baevski, Alexei; Conneau, Alexis; Auli, Michael

doi:10.21437/interspeech.2022-143

Cited by 184 publications

(69 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite being only half the size of our original model, the nine-layer sub-model significantly outperforms when evaluated on the VoxLingua107 test set. Further, its error rate surpasses the original XLS-R results [2]. Seen in Table 4, this is a result of improved performance across both short (0...5s) and long (5...20s) utterances.…”

Section: Benchmarkingmentioning

confidence: 79%

“…Alongside achieving state-of-the-art (SOTA) accuracy for automatic speech recognition (ASR), such models are notable for high performance in low-resource settings. This versatility becomes more pronounced when pre-training utilizes multilingual corpora, with fine-tuned models achieving superior accuracy in comparison to monolingual counterparts [2,3,4].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Accidental Learners: Spoken Language Identification in Multilingual Self-Supervised Models

Bartley¹,

Jia²,

Puvvada³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we extend previous self-supervised approaches for language identification by experimenting with Conformer based architecture in a multilingual pre-training paradigm. We find that pre-trained speech models optimally encode language discriminatory information in lower layers. Further, we demonstrate that the embeddings obtained from these layers are significantly robust to classify unseen languages and different acoustic environments without additional training. After fine-tuning a pre-trained Conformer model on the VoxLin-gua107 dataset, we achieve results similar to current state-ofthe-art systems for language identification. More, our model accomplishes this with 5x less parameters. We open-source the model through the NVIDIA NeMo toolkit.

show abstract

Section: Benchmarkingmentioning

confidence: 79%

Section: Introductionmentioning

confidence: 99%

Accidental Learners: Spoken Language Identification in Multilingual Self-Supervised Models

Bartley¹,

Jia²,

Puvvada³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The three data types are amenable to being transcribed using ASR, taking the burden of manual transcription off researchers' hands. Freely accessible tools like DeepSpeech (Amodei et al, 2016;Hannun et al, 2014), wav2vec2 (Baevski et al, 2020) and XLS-R (Babu et al, 2021;Baevski et al, 2020;Schneider et al, 2019) include ASR for numerous languages. New tools like cross-lingual models are also making it possible to train models for under-resourced languages.…”

Section: What Data Do We Need?mentioning

confidence: 99%

Computational sociophonetics using automatic speech recognition

Coto-Solano

2022

Language and Linguist. Compass

View full text Add to dashboard Cite

Recent years have seen numerous advances in natural language processing that can help accelerate sociophonetic work. These include software to align speech recordings with their transcriptions, as well as to transcribe audio automatically. This solves a major bottleneck and will help process larger datasets and test hypotheses more efficiently. This paper will summarise recent progress, highlight relevant examples of sociophonetic research, and comment on the technical and ethical issues at the cutting edge of natural language processing.

show abstract

“…Table VIII shows the results of replacing MFCC features with a set of publicly available neural representations: (i) multilingual bottleneck features, (ii) intermediate representations extracted from HuBERT [47], and (iii) XLS-R [48] which is a scaled up version of XLSR [49]. We include k-means as it has recently been used to cluster HuBERT features for downstream applications that require discrete representations; we set the number of k-means clusters 100 to match the truncation parameter of our Dirichlet process-based models.…”

Section: H Subspace Models With Neural Representationsmentioning

confidence: 99%

Non-Parametric Bayesian Subspace Models for Acoustic Unit Discovery

Ondel

Yusuf

Burget

et al. 2022

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

This work investigates subspace non-parametric models for the task of learning a set of acoustic units from unlabeled speech recordings. We constrain the base-measure of a Dirichlet-Process mixture with a phonetic subspace-estimated from other source languages-to build an educated prior, thereby forcing the learned acoustic units to resemble phones of known source languages. Two types of models are proposed: (i) the Subspace HMM (SHMM) which assumes that the phonetic subspace is the same for every language, (ii) the Hierarchical-Subspace HMM (H-SHMM) which relaxes this assumption and allows to have a language-specific subspace estimated on the unlabeled target data. These models are applied on 3 languages: English, Yoruba and Mboshi and they are compared with various competitive acoustic units discovery baselines. Experimental results show that both subspace models outperform other systems in terms of clustering quality and segmentation accuracy. Moreover, we observe that the H-SHMM provides results superior to the SHMM supporting the idea that language-specific priors are preferable to language-agnostic priors for acoustic unit discovery.

show abstract

XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale

Cited by 184 publications

References 35 publications

Accidental Learners: Spoken Language Identification in Multilingual Self-Supervised Models

Accidental Learners: Spoken Language Identification in Multilingual Self-Supervised Models

Computational sociophonetics using automatic speech recognition

Non-Parametric Bayesian Subspace Models for Acoustic Unit Discovery

Contact Info

Product

Resources

About