HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

Hsu, Wei-Ning; Bolte, Benjamin; Tsai, Yao-Hung Hubert; Lakhotia, Kushal; Salakhutdinov, Ruslan; Mohamed, Abdelrahman

doi:10.48550/arxiv.2106.07447

Cited by 61 publications

(142 citation statements)

References 42 publications

Supporting

Mentioning

127

Contrasting

Order By: Relevance

“…The loss weight on unmasked region does not have a large impact on the fine-tuning performance. This is different from the findings in Audio HuBERT Hsu et al (2021a), where masked prediction leads to much better performance. Given image frames as input, the prediction of cluster assignments, which are mostly determined by the accompanied audio stream, helps encode phonetic information into the visual representation.…”

Section: Modality Dropoutcontrasting

confidence: 99%

“…Our research builds on Audio HuBERT (Hsu et al, 2021a) which is a self-supervised learning framework for speech and audio. It alternates between two steps: feature clustering and masked predic-tion.…”

Section: Preliminary: Audio Hubertmentioning

confidence: 99%

“…As no ground-truth language label is given in the VoxCeleb2, we use a simple heuristic to choose the English samples. Specifically, we use an off-the-shelf character-based ASR model (Hsu et al, 2021a) trained on Librispeech which achieves 1.9%/3.5% WER on clean/other test set. We run greedy decoding on the VoxCeleb2 and use the proportion of valid English words to determine if a target utterance is English or not.…”

Section: A Detailed Experimental Setupmentioning

confidence: 99%

See 2 more Smart Citations

Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

Shi¹,

Hsu²,

Lakhotia³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Video recordings of speech contain correlated audio and visual information, providing a strong signal for speech representation learning from the speaker's lip movements and the produced sound. We introduce Audio-Visual Hidden Unit BERT (AV-HuBERT), a self-supervised representation learning framework for audio-visual speech, which masks multi-stream video input and predicts automatically discovered and iteratively refined multimodal hidden units. AV-HuBERT learns powerful audio-visual speech representation benefiting both lip-reading and automatic speech recognition. On the largest public lip-reading benchmark LRS3 (433 hours), AV-HuBERT achieves 32.5% WER with only 30 hours of labeled data, outperforming the former state-of-the-art approach (33.6%) trained with a thousand times more transcribed video data (31K hours) (Makino et al., 2019). The lip-reading WER is further reduced to 26.9% when using all 433 hours of labeled data from LRS3 and combined with self-training. Using our audio-visual representation on the same benchmark for audio-only speech recognition leads to a 40% relative WER reduction over the state-of-the-art performance (1.3% vs 2.3%). Our code and models are available at https://github.com/ facebookresearch/av_hubert

show abstract

Section: Modality Dropoutcontrasting

confidence: 99%

Section: Preliminary: Audio Hubertmentioning

confidence: 99%

Section: A Detailed Experimental Setupmentioning

confidence: 99%

See 1 more Smart Citation

Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

Shi¹,

Hsu²,

Lakhotia³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…For the regular and small models, we trained a 12-layer encoder with 2048 units, d = 256 of the hidden dimension size, and M = 4 multihead attention. The large model was trained with d = 512 and M = 8 and used HuBERT [25] features 1 pretrained on Libri-light [26].…”

Section: Methodsmentioning

confidence: 99%

Run-and-back stitch search: novel block synchronous decoding for streaming encoder-decoder ASR

Tsunoo¹,

Narisetty²,

Hentschel³

et al. 2022

Preprint

View full text Add to dashboard Cite

A streaming style inference of encoder-decoder automatic speech recognition (ASR) systems is important for reducing latency, which is essential for interactive use cases. To this end, we propose a novel blockwise synchronous decoding algorithm with a hybrid approach that combines endpoint prediction and endpoint post-determination. In the endpoint prediction, we compute the expectation of the number of tokens that are yet to be emitted in the encoder features of the current blocks using the CTC posterior. Based on the expectation value, the decoder predicts the endpoint to realize continuous block synchronization, as a running stitch. Meanwhile, endpoint post-determination probabilistically detects backward jump of the source-target attention, which is caused by the misprediction of endpoints. Then it resumes decoding by discarding those hypotheses, as back stitch. We combine these methods into a hybrid approach, namely run-and-back stitch search, which reduces the computational cost and latency. Evaluations of various ASR tasks show the efficiency of our proposed decoding algorithm, which achieves a latency reduction, for instance in the Librispeech test set from 1487 ms to 821 ms at the 90th percentile, while maintaining a high recognition accuracy.

show abstract

“…1. Specifically, by utilizing the great reconstruction ability of the vector quantized variational autoencoder (VQ-VAE) [11] or HuBERT [12] + HiFi-GAN [13] model, we transfer the target speech of speech separation model into a sequence of discrete symbols. With the estimation of discrete symbol sequence, each target speech could be re-synthesized with optional transferred style.…”

Section: Introductionmentioning

confidence: 99%

Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem

Shi¹,

Chang²,

Tomoki³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep learning based models have significantly improved the performance of speech separation with input mixtures like the cocktail party. Prominent methods (e.g., frequency-domain and time-domain speech separation) usually build regression models to predict the ground-truth speech from the mixture, using the masking-based design and the signal-level loss criterion (e.g., MSE or SI-SNR). This study demonstrates, for the first time, that the synthesis-based approach can also perform well on this problem, with great flexibility and strong potential. Specifically, we propose a novel speech separation/enhancement model based on the recognition of discrete symbols, and convert the paradigm of the speech separation/enhancement related tasks from regression to classification. By utilizing the synthesis model with the input of discrete symbols, after the prediction of discrete symbol sequence, each target speech could be resynthesized. Evaluation results based on the WSJ0-2mix and VCTKnoisy corpora in various settings show that our proposed method can steadily synthesize the separated speech with high speech quality and without any interference, which is difficult to avoid in regressionbased methods. In addition, with negligible loss of listening quality, the speaker conversion of enhanced/separated speech could be easily realized through our method.

show abstract

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

Cited by 61 publications

References 42 publications

Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

Run-and-back stitch search: novel block synchronous decoding for streaming encoder-decoder ASR

Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem

Contact Info

Product

Resources

About