Importance-Aware Information Bottleneck Learning Paradigm for Lip Reading

Abstract

Lip reading is the task of decoding text from speakers' mouth movements. Numerous deep learning-based methods have been proposed to address this task. However, these existing deep lip reading models suffer from poor generalization due to overfitting the training data. To resolve this issue, we present a novel learning paradigm that aims to improve the interpretability and generalization of lip reading models. In specific, a Variational Temporal Mask (VTM) module is customized to automatically analyze the importance of frame-level features. Furthermore, the prediction consistency constraints of global information and local temporal important features are introduced to strengthen the model generalization. We evaluate the novel learning paradigm with multiple lip reading baseline models on the LRW and LRW-1000 datasets. Experiments show that the proposed framework significantly improves the generalization performance and interpretability of lip reading models.

Multimodal audiovisual speech recognition architecture using a three‐feature multi‐fusion method for noise‐robust systems

“…Deep learning technology has recently exhibited significant performance improvements in sentence-level speech recognition, visual speech recognition, and speech-visual recognition studies using traditional prediction techniques [26][27][28][29]. When comparing the word recognition rate performance for phrases on the same benchmark dataset, the performance of deep learningbased visual recognition studies [17] improved by 36.4% over conventional visual recognition studies [30].…”