Few-Shot Adversarial Learning of Realistic Neural Talking Head Models

Abstract

SourceTarget → Landmarks → Result Source Target → Landmarks → Result Figure 1: The results of talking head image synthesis using face landmark tracks extracted from a different video sequence of the same person (on the left), and using face landmarks of a different person (on the right). The results are conditioned on the landmarks taken from the target frame, while the source frame is an example from the training set. The talking head models on the left were trained using eight frames, while the models on the right were trained in a one-shot manner. AbstractSeveral recent works have shown how highly realistic human head images can be obtained by training convolutional neural networks to generate them. In order to create a personalized talking head model, these works require training on a large dataset of images of a single person. However, in many practical scenarios, such personalized talking head models need to be learned from a few image views of a person, potentially even a single image. Here, we present a system with such few-shot capability. It performs lengthy meta-learning on a large dataset of videos, and after that is able to frame few-and one-shot learning of neural talking head models of previously unseen people as adversarial training problems with high capacity generators and discriminators. Crucially, the system is able to initialize the parameters of both the generator and the discriminator in a person-specific way, so that training can be based on just a few images and done quickly, despite the need to tune tens of millions of parameters. We show that such an approach is able to learn highly realistic and personalized talking head models of new people and even portrait paintings.

Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos

“…This work greatly simplified the requirements for attackers: sim-ply acquire a picture of the victim (usually a profile picture on a social network or an ID photo). Zakharov et al [35] followed up by improving the quality of videos generated using only a few input images. Vougioukas et al [36] raised the bar by introducing a method for animating a facial image from an audio track containing speech.…”

A State-of-the-Art Review on Image Synthesis With Generative Adversarial Networks

“…Most of these applications involve image processing. Although there have been some studies involving video processing, such as video generation [115], video colorization [116], [117], video inpainting [118], motion transfer [119], and facial animation synthesis [120]- [123], the research on video using GANs is limited. In addition, although GANs have been applied to the generation and synthesis of 3D models, such as 3D colorization [124], 3D face reconstruction [125], [126], 3D character animation [127], and 3D textured object generation [128], the results are far from perfect.…”

Social and Governance Implications of Improved Data Efficiency

“…In fact, the risk from these systems can be understood as a product of their high data efficiency. Deepfakes that required 1000s of hours of video would be much less disruptive, whereas a recent system was able to base deepfakes on as few as 32 video frames [35]. Data efficiency is a crucial parameter in judging misuse potential.…”