Adaptive Semantic-Spatio-Temporal Graph Convolutional Network for Lip Reading

Abstract

The goal of this work is to recognize words, phrases, and sentences being spoken by a talking face without given the audio. Current deep learning approaches for lip reading focus on exploring the appearance and optical flow information of videos. However, these methods do not fully exploit the characteristics of lip motion. In addition to appearance and optical flow, the mouth contour deformation usually conveys significant information that is complementary to others. However, the modeling of dynamic mouth contour has received little attention than that of appearance and optical flow. In this work, we propose a novel model of dynamic mouth contours called Adaptive Semantic-Spatio-Temporal Graph Convolution Network (ASST-GCN), to go beyond previous methods by automatically learning both the spatial and temporal information from videos. To combine the complementary information from appearance and mouth contour, a two-stream visual front-end network is proposed. Experimental results demonstrate that the proposed method significantly outperforms the state-of-the-art lip reading methods on several large-scale lip reading benchmarks.

Deep Learning for Visual Speech Analysis: A Survey

“…Recently, Fenghour et al [43] conducted a survey reviewing deep learning driven VSR methods, including audio-visual datasets, feature extraction, classification networks and classification schemes. However, some essential advances of VSR were omitted, such as self-supervised learning methods [47,48,49,50], cross-modal knowledge distillation methods [34,51,52], graph neural networks backbone architectures [53,54], etc. Chen et al [42] conducted a thoughtful analysis across several representative identity-independent VSG methods and designed a performance evaluation benchmark for VSG.…”

“…Among them, mouth-centered videos and dense optical flow are regular grid data, so CNNs are the most suitable and commonly used backbone architectures for them. On the other hand, as landmark points are irregular data, some existing works [53,54,118] adopted Graph Convolution Networks (GCNs) to extract visual features from landmark points. Next, we review these backbone architectures.…”

“…However, the lip graph connection relations do not naturally exist, and the intuition-guided predefined lip graph restricts the representation ability of shape-based features. Motivated by this, Sheng et al [54] proposed an Adaptive Semantic-Spatial-Temporal Graph Convolution Network (ASST-GCN). Unlike [53], the ASST-GCN parameterizes Since the complementarity between image and landmark features, the combination of CNNs and GCNs is often widely adopted [53,54,118].…”

“…Motivated by this, Sheng et al [54] proposed an Adaptive Semantic-Spatial-Temporal Graph Convolution Network (ASST-GCN). Unlike [53], the ASST-GCN parameterizes Since the complementarity between image and landmark features, the combination of CNNs and GCNs is often widely adopted [53,54,118].…”

IR-UWB Radar-Based Contactless Silent Speech Recognition of Vowels, Consonants, Words, and Phrases

“…Surface electromyography [16], [17], vision [18], [19], [20], [21], [22], [23], ultrasound imaging [24], [25], and radar [26], [27], [28], [29], [30], [31] are techniques for capturing nonacoustic speech-related biosignals without the need to place sensors inside the oral cavity. Although these techniques are more convenient than the aforementioned ones, they have some shortcomings.…”