LiRA: Learning Visual Speech Representations from Audio Through Self-Supervision

Ma, Pingchuan; Mira, Rodrigo; Petridis, Stavros; Schuller, Björn W.; Pantić, Maja

doi:10.21437/interspeech.2021-1360

Cited by 28 publications

(7 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, good audio representations can be learned by predicting handcrafted audio features [73] or by using joint audio and visual supervision [74]. Similarly, visual speech representations can be learned by predicting audio features [75]. Hence, the proposed auxiliary task provides additional supervision to the intermediate layers of the model which in turns results in better visual representations and improved performance.…”

Section: Prediction-based Auxiliary Tasksmentioning

confidence: 99%

Visual Speech Recognition for Multiple Languages in the Wild

Ma¹,

Petridis²,

Pantić³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Visual speech recognition (VSR) aims to recognise the content of speech based on the lip movements without relying on the audio stream. Advances in deep learning and the availability of large audio-visual datasets have led to the development of much more accurate and robust VSR models than ever before. However, these advances are usually due to larger training sets rather than the model design. In this work, we demonstrate that designing better models is equally important to using larger training sets. We propose the addition of prediction-based auxiliary tasks to a VSR model and highlight the importance of hyper-parameter optimisation and appropriate data augmentations.We show that such model works for different languages and outperforms all previous methods trained on publicly available datasets by a large margin. It even outperforms models that were trained on non-publicly available datasets containing up to to 21 times more data. We show furthermore that using additional training data, even in other languages or with automatically generated transcriptions, results in further improvement.

show abstract

Section: Prediction-based Auxiliary Tasksmentioning

confidence: 99%

Visual Speech Recognition for Multiple Languages in the Wild

Ma¹,

Petridis²,

Pantić³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…[21][22][23]) and also produces features that separate suprasegmental properties such as speaker identities [17]. However, to the best of our knowledge, [24] is the only study so far using CPC for AL.…”

Section: Methodsmentioning

confidence: 99%

Analysis of Self-Supervised Learning and Dimensionality Reduction Methods in Clustering-Based Active Learning for Speech Emotion Recognition

Vaaras¹,

Airaksinen²,

Räsänen³

2022

Interspeech 2022

View full text Add to dashboard Cite

When domain experts are needed to perform data annotation for complex machine-learning tasks, reducing annotation effort is crucial in order to cut down time and expenses. For cases when there are no annotations available, one approach is to utilize the structure of the feature space for clustering-based active learning (AL) methods. However, these methods are heavily dependent on how the samples are organized in the feature space and what distance metric is used. Unsupervised methods such as contrastive predictive coding (CPC) can potentially be used to learn organized feature spaces, but these methods typically create highdimensional features which might be challenging for estimating data density. In this paper, we combine CPC and multiple dimensionality reduction methods in search of functioning practices for clustering-based AL. Our experiments for simulating speech emotion recognition system deployment show that both the local and global topology of the feature space can be successfully used for AL, and that CPC can be used to improve clustering-based AL performance over traditional signal features. Additionally, we observe that compressing data dimensionality does not harm AL performance substantially, and that 2-D feature representations achieved similar AL performance as higher-dimensional representations when the number of annotations is not very low.

show abstract

“…Initialisation To investigate the impact of initialisation we consider three cases: 1) we train the model from scratch using only the LRW training set, 2) we pre-train the encoder from Fig. 1 on the LRS3 dataset [17] using the LiRA [12] selfsupervised approach and fine-tune it on the LRW training set.…”

Section: Self-distillation Modelsmentioning

confidence: 99%

“…"Scratch" denotes a model trained from scratch without using external data. "LiRA(LRS3)" indicates a self-supervised pre-trained model using LiRA[12] on the LRS3 dataset, and "LRS2&3+AVS" indicates a fully supervised pre-trained model on LRS2, LRS3 and AVSpeech.…”

mentioning

confidence: 99%

Training Strategies for Improved Lip-Reading

Wang

Petridis

et al. 2022

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Several training strategies and temporal models have been recently proposed for isolated word lip-reading in a series of independent works. However, the potential of combining the best strategies and investigating the impact of each of them has not been explored. In this paper, we systematically investigate the performance of state-of-the-art data augmentation approaches, temporal models and other training strategies, like self-distillation and using word boundary indicators. Our results show that Time Masking (TM) is the most important augmentation followed by mixup and Densely-Connected Temporal Convolutional Networks (DC-TCN) are the best temporal model for lip-reading of isolated words. Using selfdistillation and word boundary indicators is also beneficial but to a lesser extent. A combination of all the above methods results in a classification accuracy of 93.4%, which is an absolute improvement of 4.6% over the current state-of-theart performance on the LRW dataset. The performance can be further improved to 94.1% by pre-training on additional datasets. An error analysis of the various training strategies reveals that the performance improves by increasing the classification accuracy of hard-to-recognise words.

show abstract

LiRA: Learning Visual Speech Representations from Audio Through Self-Supervision

Cited by 28 publications

References 25 publications

Visual Speech Recognition for Multiple Languages in the Wild

Visual Speech Recognition for Multiple Languages in the Wild

Analysis of Self-Supervised Learning and Dimensionality Reduction Methods in Clustering-Based Active Learning for Speech Emotion Recognition

Training Strategies for Improved Lip-Reading

Contact Info

Product

Resources

About