Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers

Abstract

Lip reading has witnessed unparalleled development in recent years thanks to deep learning and the availability of largescale datasets. Despite the encouraging results achieved, the performance of lip reading, unfortunately, remains inferior to the one of its counterpart speech recognition, due to the ambiguous nature of its actuations that makes it challenging to extract discriminant features from the lip movement videos. In this paper, we propose a new method, termed as Lip by Speech (LIBS), of which the goal is to strengthen lip reading by learning from speech recognizers. The rationale behind our approach is that the features extracted from speech recognizers may provide complementary and discriminant clues, which are formidable to be obtained from the subtle movements of the lips, and consequently facilitate the training of lip readers. This is achieved, specifically, by distilling multigranularity knowledge from speech recognizers to lip readers. To conduct this cross-modal knowledge distillation, we utilize an efficacious alignment scheme to handle the inconsistent lengths of the audios and videos, as well as an innovative filtering strategy to refine the speech recognizer's prediction. The proposed method achieves the new state-of-the-art performance on the CMLR and LRS2 datasets, outperforming the baseline by a margin of 7.66% and 2.75% in character error rate, respectively.

FastLR: Non-Autoregressive Lipreading Model with Integrate-and-Fire

“…Note that experiment on GRID dataset needs more training steps, since it is trained with its visual frontend together from scratch, different from experiments on LRS2 dataset. Moreover, the first 45k steps in warm-up stage for LRS2 are trained on LRS2-pretrain sub-dataset and all the left steps are trained on LRS2-main sub-dataset [1,2,33].…”

“…The former surpasses the performance of all previous work on LRS2-BBC dataset by a large margin. To boost the performance of lipreading, Petridis et al [19] present a hybrid CTC/Attention architecture aiming to obtain the better alignment than attention-only mechanism, Zhao et al [33] provide the idea that transferring knowledge from audio-speech recognition model to lipreading model by distillation.…”

“…And we make use of the pre-train dataset provided by LRS2 which contains 96k sentences for pretraining. Following previous works [1,2,33], the input video frames are converted to grey scale and centrally cropped into 114 × 114 images. As for the text sentence, we split each word token into subwords using BPE [23], and set the vocabulary size to 1k considering the vocabulary size of LRS2.…”

“…It is worth noting that, training the visual front-end together with the main model could obtain poor results on LRS2, which is observed in previous works [1]. Thus, as Zhao et al [33] do, we utilize the frozen visual front-end provided by Afouras et al [1], which is pretrained on a non-public datasets MV-LRS [8], to exact the visual features. And then, we train FastLR on these features end-to-end.…”

“…Existing methods mainly adopt autoregressive (AR) model, either based on RNN [25,33], or Transformer [1,2]. Those systems generate each target token conditioned on the sequence of tokens generated previously, which hinders the parallelizability.…”