Talking Face Generation by Adversarially Disentangled Audio-Visual Representation

Abstract

Talking face generation aims to synthesize a sequence of face images that correspond to a clip of speech. This is a challenging task because face appearance variation and semantics of speech are coupled together in the subtle movements of the talking face regions. Existing works either construct specific face appearance model on specific subjects or model the transformation between lip motion and speech. In this work, we integrate both aspects and enable arbitrary-subject talking face generation by learning disentangled audio-visual representation. We find that the talking face sequence is actually a composition of both subject-related information and speech-related information. These two spaces are then explicitly disentangled through a novel associative-and-adversarial training process. This disentangled representation has an advantage where both audio and video can serve as inputs for generation. Extensive experiments show that the proposed approach generates realistic talking face sequences on arbitrary subjects with much clearer lip motion patterns than previous work. We also demonstrate the learned audio-visual representation is extremely useful for the tasks of automatic lip reading and audio-video retrieval.

Towards Automatic Face-to-Face Translation

“…Algorithm PSNR SSIM LMD Chung et al [8] 28.06 0.460 2.22 Zhou et al [35] 26.80 0.884 -LipGAN (Ours) 33.4 0.960 0.60 Table 5: Our proposed LipGAN model achieves significant improvements over existing competitive approaches across all standard quantitative metrics.…”

“…This leads them to use a simple fully convolutional encoder-decoder model. Even more recently, a different solution to the problem was proposed by Zhou et al [35], in which they use audio-visual speech recognition as a probe task for associating audio-visual representations, and then employ adversarial learning to disentangle the subject-related and speech-related information inside them. However, we observed two major limitations in their work.…”

“…In contrast, our LipGAN directly optimizes for improved lip-sync by employing an adversarial network that measures the extent of lip-sync between the frames generated by the generator and the corresponding audio sample. Additionally, both Zhou et al [35] and Chung et al [8] normalize the pose of the input faces to a canonical pose, thus making it difficult to blend the generated faces in the original input video. Proposed LipGAN tackles this problem by providing additional information about the pose of the target face as an input to the model thus making the final blending of the generated face in the target video fairly straightforward.…”

“…Smooth blending into target video Suwajanakorn et al [29] × × ✓ ✓ Kumar et al [17] × × × ✓ Zhou et al [35] ✓ × × × Chung et al [8] ✓…”

“…While using only an L2 reconstruction loss for the generator can generate satisfactory talking faces [8], employing strong additional supervision can help the generator learn robust, accurate phonemeviseme mappings and also make the facial movements more natural. Zhou et al [35] employed audio-visual speech recognition as a probe task to associate the acoustic and visual information. However, this makes the setup language-specific and offers only indirect supervision.…”