Audio-Driven Talking Face Video Generation With Dynamic Convolution Kernels

Ye, Zipeng; Xia, Mengfei; Ran, Yi; Zhang, Jie; Lai, Yuekun; Huang, Xuwei; Zhang, Guoxin; Liu, Yong-Jin

doi:10.1109/tmm.2022.3142387

Cited by 39 publications

(14 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As shown in Fig. 8(l), unlike the previous concatenation-based feature fusion strategy, Ye et al [74] presented a full convolutional neural network with dynamic convolution kernels (DCKs) for crossmodal feature fusion, which extracts features from audio and reshapes features as DCKs of the fully convolutional network. Due to the simple yet effective network architecture, the realtime performance of VSG is significantly improved.…”

Section: Other Methodsmentioning

confidence: 99%

Deep Learning for Visual Speech Analysis: A Survey

Sheng¹,

Kuang²,

Bai³

et al. 2022

Preprint

View full text Add to dashboard Cite

Visual speech, referring to the visual domain of speech, has attracted increasing attention due to its wide applications, such as public security, medical treatment, military defense, and film entertainment. As a powerful AI strategy, deep learning techniques have extensively promoted the development of visual speech learning. Over the past five years, numerous deep learning based methods have been proposed to address various problems in this area, especially automatic visual speech recognition and generation. To push forward future research on visual speech, this paper aims to present a comprehensive review of recent progress in deep learning methods on visual speech analysis. We cover different aspects of visual speech, including fundamental problems, challenges, benchmark datasets, a taxonomy of existing methods, and state-of-the-art performance. Besides, we also identify gaps in current research and discuss inspiring future research directions.

show abstract

Section: Other Methodsmentioning

confidence: 99%

Deep Learning for Visual Speech Analysis: A Survey

Sheng¹,

Kuang²,

Bai³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Other methods rely on audio inputs to control the lower part of the face (lips, jaw movement) [5,16,28] to high-quality facial images while maintaining control over expression, illumination, and pose. However, Sty-leRig fails to exploit 3DMM's full expression space, resulting in incorrect expression mappings for the final result (e.g.…”

Section: Related Workmentioning

confidence: 99%

Real-Time Drift-Driving Control for an Autonomous Vehicle: Learning from Nonlinear Model Predictive Control via a Deep Neural Network

et al. 2022

View full text Add to dashboard Cite

A drift-driving maneuver is a control technique used by an expert driver to control a vehicle along a sharply curved path or slippery road. This study develops a nonlinear model predictive control (NMPC) method for the autonomous vehicle to perform a drift maneuver and generate the datasets necessary for training the deep neural network(DNN)-based drift controller. In general, the NMPC method is based on numerical optimization which is difficult to run in real-time. By replacing the previously designed NMPC method with the proposed DNN-based controller, we avoid the need for complex numerical optimization of the vehicle control, thereby reducing the computational load. The performance of the developed data-driven drift controller is verified through realistic simulations that included drift scenarios. Based on the results of the simulations, the DNN-based controller showed similar tracking performance to the original nonlinear model predictive controller; moreover, the DNN-based controller can demonstrate stable computation time, which is very important for the safety critical control objective such as drift maneuver.

show abstract

“…Therefore, using different static neural textures can represent different expressions and dynamic neural textures can be regarded as an approximation for a set of static neural textures. Inspired by dynamic convolution kernels [34], we can understand dynamic neural textures in the following way. Denote by E the space of all expressions.…”

Section: Dynamic Neural Texturesmentioning

confidence: 99%

Dynamic Neural Textures: Generating Talking-Face Videos with Continuously Controllable Expressions

Ye¹,

Sun²,

Wen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, talking-face video generation has received considerable attention. So far most methods generate results with neutral expressions or expressions that are implicitly determined by neural networks in an uncontrollable way. In this paper, we propose a method to generate talking-face videos with continuously controllable expressions in real-time. Our method is based on an important observation: In contrast to facial geometry of moderate resolution, most expression information lies in textures. Then we make use of neural textures to generate high-quality talking face videos and design a novel neural network that can generate neural textures for image frames (which we called dynamic neural textures) based on the input expression and continuous intensity expression coding (CIEC). Our method uses 3DMM as a 3D model to sample the dynamic neural texture. The 3DMM does not cover the teeth area, so we propose a teeth submodule to complete the details in teeth.Results and an ablation study show the effectiveness of our method in generating high-quality talking-face videos with continuously controllable expressions. We also set up four baseline methods by combining existing representative methods and compare them with our method. Experimental results including a user study show that our method has the best performance. CCS CONCEPTS• Information systems → Multimedia content creation; • Computing methodologies → Appearance and texture representations; Neural networks; Image-based rendering.

show abstract

Audio-Driven Talking Face Video Generation With Dynamic Convolution Kernels

Cited by 39 publications

References 38 publications

Deep Learning for Visual Speech Analysis: A Survey

Deep Learning for Visual Speech Analysis: A Survey

Real-Time Drift-Driving Control for an Autonomous Vehicle: Learning from Nonlinear Model Predictive Control via a Deep Neural Network

Dynamic Neural Textures: Generating Talking-Face Videos with Continuously Controllable Expressions

Contact Info

Product

Resources

About