Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss

Abstract

We devise a cascade GAN approach to generate talking face video, which is robust to different face shapes, view angles, facial characteristics, and noisy audio conditions. Instead of learning a direct mapping from audio to video frames, we propose first to transfer audio to high-level structure, i.e., the facial landmarks, and then to generate video frames conditioned on the landmarks. Compared to a direct audio-to-image approach, our cascade approach avoids fitting spurious correlations between audiovisual signals that are irrelevant to the speech content. We, humans, are sensitive to temporal discontinuities and subtle artifacts in video. To avoid those pixel jittering problems and to enforce the network to focus on audiovisual-correlated regions, we propose a novel dynamically adjustable pixel-wise loss with an attention mechanism. Furthermore, to generate a sharper image with well-synchronized facial movements, we propose a novel regression-based discriminator structure, which considers sequence-level information along with frame-level information. Thoughtful experiments on several datasets and realworld samples demonstrate significantly better results obtained by our method than the state-of-the-art methods in both quantitative and qualitative comparisons.

Talking Face Generation with Expression-Tailored Generative Adversarial Network

“…In this section, we compare ET-GAN with the state-of-the-art methods including Speech2Vid [23], DAVS [49], ATVG [8] and SDA [42] using metrics introduced above. As shown in Table 1, ET-GAN surpasses other state-of-the-art methods from quantitative aspect, which means ET-GAN has higher fidelity than others.…”

“…As shown in Figure 6, these continuous video frames come from the same identity saying the same word with different methods. Images in the first row are ground-truths, and the rest rows are state-of-thearts [8,23,42,49] and ET-GAN. Faces generated by Speech2Vid and ATVG barely contain hairs and necks.…”

“…Landmark is important for guiding facial videos generation [29,34,43]. ATVG [8] utilized facial landmarks to assist generation, and exploited a dynamically adjustable pixel-wise loss along with an attention mechanism. DAVS [49] indicated that identity information can also be extract from videos rather than images.…”

“…Synthesizing realistic talking faces with expression that exactly match certain speech is challenging for three reasons. First, the acuteness of human visual system leads to low tolerance to modeling mistakes in facial videos, so expression-absent talking faces are not acceptable [8]. Second, audio-video synchronization is strictly required, so learning facial variation and speech vibrations is important for better audio-visual matching [49].…”

A State-of-the-Art Review on Image Synthesis With Generative Adversarial Networks

“…Most of these applications involve image processing. Although there have been some studies involving video processing, such as video generation [115], video colorization [116], [117], video inpainting [118], motion transfer [119], and facial animation synthesis [120]- [123], the research on video using GANs is limited. In addition, although GANs have been applied to the generation and synthesis of 3D models, such as 3D colorization [124], 3D face reconstruction [125], [126], 3D character animation [127], and 3D textured object generation [128], the results are far from perfect.…”