mSLAM: Massively multilingual joint pre-training for speech and text

Bapna, Ankur; Cherry, Colin; Jia, Yali; Johnson, Melvin; Yang, Cheng; Khanuja, Simran; Riesa, Jason; Conneau, Alexis

doi:10.48550/arxiv.2202.01374

Cited by 12 publications

(24 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(Right) SoTA results on public speech translation tasks. Results presented are presented as high/middle/low resources languages defined in [20]. Higher is better.…”

Section: Key Findingsmentioning

confidence: 99%

BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition

Park

Han

Qin

et al. 2022

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages, and fine-tuning on a smaller labeled dataset. We use multilingual pre-training with random-projection quantization and speech-text modality matching to achieve state-of-the-art performance on downstream multilingual ASR and speech-to-text translation tasks. We also demonstrate that despite using a labeled training set 1/7-th the size of that used for the Whisper model [1], our model exhibits comparable or better performance on both in-domain and out-of-domain speech recognition tasks across many languages.

show abstract

“…(Right) SoTA results on public speech translation tasks. Results presented are presented as high/middle/low resources languages defined in [20]. Higher is better.…”

Section: Key Findingsmentioning

confidence: 99%

BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition

Park

Han

Qin

et al. 2022

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

show abstract

“…We can improve upon their method through a combination of speech-to-text ALMs together with LM-based commonsense reasoning. First, we transcribe the audio from all videos with speech-to-text ALMs (Bapna et al 2022) (also called ASR, or automatic speech recognition), using the publicly-available Google Cloud speech-to-text API. 5 Although raw transcripts may be challenging to incorporate into meaningful improvements for video/caption retrieval, we may leverage reasoning capabilities from large LMs in order to usefully harness the transcripts.…”

Section: System Overview: Socratic Video-to-text Retrievalmentioning

confidence: 99%

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

Zeng¹,

Wong²,

Welker³

et al. 2022

Preprint

View full text Add to dashboard Cite

Large foundation models can exhibit unique capabilities depending on the domain of data they are trained on. While these domains are generic, they may only barely overlap. For example, visual-language models (VLMs) are trained on Internet-scale image captions, but large language models (LMs) are further trained on Internetscale text with no images (e.g. from spreadsheets, to SAT questions). As a result, these models store different forms of commonsense knowledge across different domains. In this work, we show that this model diversity is symbiotic, and can be leveraged to build AI systems with structured Socratic dialogue -in which new multimodal tasks are formulated as a guided languagebased exchange between different pre-existing foundation models, without additional finetuning. In the context of egocentric perception, we present a case study of Socratic Models (SMs) that can provide meaningful results for complex tasks such as generating freeform answers to contextual questions about egocentric video, by formulating video Q&A as short story Q&A, i.e. summarizing the video into a short story, then answering questions about it. Additionally, SMs can generate captions for Internet images, and are competitive with state-of-the-art on zero-shot video-to-text retrieval with 42.8 R@1 on MSR-VTT 1k-A. SMs demonstrate how to compose foundation models zeroshot to capture new multimodal functionalities, without domain-specific data collection. Prototypes are available at socraticmodels.github.io.

show abstract

“…Recently, self-supervised pre-training of large transformer encoders on massive amounts of unlabeled audio data followed by task-specific fine-tuning has emerged as the de-facto approach for achieving state-of-the-art performance on several tasks in spoken language processing. However, popular selfsupervised representation learning (SSL) approaches such as Wav2vec-2.0 [1] and others [2]- [12] learn speech embedding at acoustic frame-level, i.e., for short speech segments of duration 10 to 20 milliseconds.…”

Section: Introductionmentioning

confidence: 99%

SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation

Khurana¹,

Laurent²,

Glass³

2022

Preprint

View full text Add to dashboard Cite

We propose the SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation learning framework. Unlike previous works on speech representation learning, which learns multilingual contextual speech embedding at the resolution of an acoustic frame (10-20ms), this work focuses on learning multimodal (speech-text) multilingual speech embedding at the resolution of a sentence (5-10s) such that the embedding vector space is semantically aligned across different languages. We combine state-of-the-art multilingual acoustic frame-level speech representation learning model XLS-R with the Language Agnostic BERT Sentence Embedding (LaBSE) model to create an utterance-level multimodal multilingual speech encoder SAMU-XLSR. Although we train SAMU-XLSR with only multilingual transcribed speech data, cross-lingual speech-text and speechspeech associations emerge in its learned representation space. To substantiate our claims, we use SAMU-XLSR speech encoder in combination with a pre-trained LaBSE text sentence encoder for cross-lingual speech-to-text translation retrieval, and SAMU-XLSR alone for cross-lingual speech-to-speech translation retrieval. We highlight these applications by performing several cross-lingual text and speech translation retrieval tasks across several datasets.

show abstract

mSLAM: Massively multilingual joint pre-training for speech and text

Cited by 12 publications

References 0 publications

BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition

BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation

Contact Info

Product

Resources

About