Deep Learning for Visual Speech Analysis: A Survey

Sheng, Changchong; Kuang, Gangyao; Bai, Liang; Hou, Chenping; Guo, Yinglu; Xu, Xin; Pietikäinen, Matti; Liu, Li

doi:10.48550/arxiv.2205.10839

Cited by 6 publications

(10 citation statements)

References 145 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, some other reviews, such as [73,79], cover more recent datasets. However, these works provide only a combined list of datasets in one table and focus mainly on lip-reading datasets rather than on AV datasets.…”

Section: Recent Audio-visual Speech Datasetsmentioning

confidence: 99%

“…), the development of AVSR methodology is still at an early stage and does not yet meet the performance standards required for practical implementation in real-world applications. This is certainly not due to a lack of effort on the part of researchers, as there have been many excellent works on AVSR [73]. Therefore, a comprehensive analysis of recent advances, identification of key barriers and unresolved issues, and exploration of potential avenues for future research are essential.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Review of Recent Advances on Deep Learning Methods for Audio-Visual Speech Recognition

2023

View full text Add to dashboard Cite

This article provides a detailed review of recent advances in audio-visual speech recognition (AVSR) methods that have been developed over the last decade (2013–2023). Despite the recent success of audio speech recognition systems, the problem of audio-visual (AV) speech decoding remains challenging. In comparison to the previous surveys, we mainly focus on the important progress brought with the introduction of deep learning (DL) to the field and skip the description of long-known traditional “hand-crafted” methods. In addition, we also discuss the recent application of DL toward AV speech fusion and recognition. We first discuss the main AV datasets used in the literature for AVSR experiments since we consider it a data-driven machine learning (ML) task. We then consider the methodology used for visual speech recognition (VSR). Subsequently, we also consider recent AV methodology advances. We then separately discuss the evolution of the core AVSR methods, pre-processing and augmentation techniques, and modality fusion strategies. We conclude the article with a discussion on the current state of AVSR and provide our vision for future research.

show abstract

Section: Recent Audio-visual Speech Datasetsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A Review of Recent Advances on Deep Learning Methods for Audio-Visual Speech Recognition

2023

View full text Add to dashboard Cite

show abstract

“…In this introduction, we provide an overview of the motivation behind leveraging deep learning for lip reading, highlighting the challenges faced by traditional methods and the potential of deep learning approaches to overcome these challenges. [4] We also outline the objectives of our proposed approach and the structure of the paper. Through this research, we aim to contribute to the advancement of lip-reading technology, with potential applications in communication aids, accessibility solutions, and human-computer interaction systems.…”

Section: Introductionmentioning

confidence: 99%

Enhancing Lip Reading: A Deep Learning Approach with CNN and RNN Integration

Rutuja Rajendra Patil, Devika A. Verma

2024

jes

View full text Add to dashboard Cite

This research introduces an innovative approach to enhance lip reading-based text extraction and translation through the integration of a double Convolutional Neural Network (CNN) coupled with Recurrent Neural Network (RNN) architecture. The proposed model aims to leverage the strengths of both CNN and RNN to achieve superior accuracy in lip movement interpretation and subsequent text extraction. The methodology involves training the double CNN+RNN model on extensive datasets containing synchronized lip movements and corresponding linguistic expressions. The initial layers of the model utilize CNNs to effectively capture spatial features from the visual input of lip images. The extracted features are then fed into RNN layers, allowing the model to grasp temporal dependencies and contextual information crucial for accurate lip reading. The trained model showcases its proficiency in extracting textual content from spoken words, demonstrating an advanced capability to decipher nuances in lip gestures. Furthermore, the extracted text undergoes a translation process, enabling the conversion of spoken language into various target languages. This research not only contributes to the advancement of lip reading technologies but also establishes a robust foundation for real-world applications such as accessibility solutions for individuals with hearing impairments, real-time multilingual translation services, and improved communication in challenging acoustic environments. The abstract concludes with a discussion on the potential impact of the double CNN+RNN model in pushing the boundaries of human-computer interaction, emphasizing the synergy between deep learning, lip reading, and translation technologies

show abstract

“…However, current methods mainly focus on improving the synchronization between lip movements and speech [38], neglecting the emotional variation of facial expressions. We argue that emotions are an essential aspect of human communication and expression, and emotion absence in 3D facial animations may cause the uncanny valley effect.…”

Section: Introductionmentioning

confidence: 99%

EmoTalk: Speech-driven emotional disentanglement for 3D face animation

Peng¹,

Wu²,

Xu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Speech-driven 3D face animation aims to generate realistic facial expressions that match the speech content and emotion. However, existing methods often neglect emotional facial expressions or fail to disentangle them from speech content. To address this issue, this paper proposes an end-to-end neural network to disentangle different emotions in speech so as to generate rich 3D facial expressions. Specifically, we introduce the emotion disentangling encoder (EDE) to disentangle the emotion and content in the speech by cross-reconstructed speech signals with different emotion labels. Then an emotion-guided feature fusion decoder is employed to generate a 3D talking face with enhanced emotion. The decoder is driven by the disentangled identity, emotional, and content embeddings so as to generate controllable personal and emotional styles. Finally, considering the scarcity of the 3D emotional talking face data, we resort to the supervision of facial blendshapes, which enables the reconstruction of plausible 3D faces from 2D emotional data, and contribute a large-scale 3D emotional talking face dataset (3D-ETF) to train the network. Our experiments and user studies demonstrate that our approach outperforms state-of-the-art methods and exhibits more diverse facial movements. We recommend watching the supplementary video: https://ziqiaopeng. github.io/emotalk

show abstract

Deep Learning for Visual Speech Analysis: A Survey

Cited by 6 publications

References 145 publications

A Review of Recent Advances on Deep Learning Methods for Audio-Visual Speech Recognition

A Review of Recent Advances on Deep Learning Methods for Audio-Visual Speech Recognition

Enhancing Lip Reading: A Deep Learning Approach with CNN and RNN Integration

EmoTalk: Speech-driven emotional disentanglement for 3D face animation

Contact Info

Product

Resources

About