FSGAN: Subject Agnostic Face Swapping and Reenactment

Abstract

Figure 1: Face swapping and reenactment. Left: Source face swapped onto target. Right: Target video used to control the expressions of the face appearing in the source image. In both cases, our results appears in the middle. For more information please visit our website: https://nirkin.com/fsgan. AbstractWe present Face Swapping GAN (FSGAN) for face swapping and reenactment. Unlike previous work, FSGAN is subject agnostic and can be applied to pairs of faces without requiring training on those faces. To this end, we describe a number of technical contributions. We derive a novel recurrent neural network (RNN)-based approach for face reenactment which adjusts for both pose and expression variations and can be applied to a single image or a video sequence. For video sequences, we introduce continuous interpolation of the face views based on reenactment, Delaunay Triangulation, and barycentric coordinates. Occluded face regions are handled by a face completion network. Finally, we use a face blending network for seamless blending of the two faces while preserving target skin color and lighting conditions. This network uses a novel Poisson blending loss which combines Poisson optimization with perceptual loss. We compare our approach to existing state-of-the-art systems and show our results to be both qualitatively and quantitatively superior.

SimSwap

“…However, owing to the limited expressiveness of the 3D face dataset, methods [4,27] replying on 3D models often failed to reproduce the expressions accurately. Recently, FSGAN [26] proposed a two-stage architecture which first conduct the expression and posture transfer with a face reenactment network and then used another face inpainting network to blend the source face into the target image. A common problem for source-oriented methods is that they are sensitive to the input source image.…”

Fighting Deepfakes: Media and Internet Giants’ Converging and Diverging Strategies Against Hi-Tech Misinformation

“…Therefore, future creations will be improved thanks to past experiences. This feature makes this misinformation procedure more dangerous, especially due to the emergence of mobile apps and computer programmes that allow users without computer programming training to produce deepfakes (Nirkin, Keller, & Hassner, 2019;Schwartz, 2018). Farid et al (2019, pp.…”

VR content creation and exploration with deep learning: A survey

“…To keep structural consistency of faces in face reenactment, Wu et al [168] proposed a method that first maps the source face onto a boundary latent space, then transforms the source boundary to adapt to the target boundary, and finally decodes the transformed boundary to generate the reenacted face. The Face Swapping GAN (FSGAN) was proposed to swap and reenact faces in a subject agnostic manner [169]. An RNN-based approach which adjusts for pose and expression variations was proposed, assisted by a face completion network and a face blending network to generate realistic face swapping results.…”