High-resolution Face Swapping via Latent Semantics Disentanglement

Xu, Yangyang; Deng, Bailin; Wang, Junle; Jing, Yanqing; Pan, Jia; He, Shengfeng

doi:10.1109/cvpr52688.2022.00749

Cited by 55 publications

(43 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although this line of work has achieved satisfying performance in identity transferring and attribute preserving, the resolution of the face swapping images is still limited. 3) To generate high-resolution swapped faces, GAN inversion based face swapping methods are proposed [3], [4], [34]. For example, MegaFS [3] uses the pre-trained StyleGAN2 [35] as the decoder and trains the encoder only.…”

Section: Related Workmentioning

confidence: 99%

“…In conclusion, the face swapping images combines the identity features of their source and target faces instead of completely maintaining the source identity as expected. We attribute the phenomenon to the entanglement of the facial attributes and identity features [4], [26]. As it's difficult to disentangle attributes and identities, the identity of the target face will inevitably exist in the swapped face.…”

Section: Identity Combining Nature Of Face Swappingmentioning

confidence: 99%

“…F ACE forgery technologies are constantly evolving and becoming increasingly realistic. One major category of face forgery is face swapping [1], [2], [3], [4], [5], which aims at replacing the face of one person in an image or a video with the face of another person [6]. Despite its applications in fields like entertainment, face swapping can cause serious social risks.…”

Section: Introductionmentioning

confidence: 99%

“…The key idea is that the identity of the swapped face is a combination of the two identities involved in face swapping, resulting in more confusion for the face identification models. Such identity combination results from the basic demand of face swapping, i.e., preserving visual features of both faces, as well as the great challenge to completely disentangle identity with other facial attributes [4], [26]. And we measure how confusing one face image is to the model with the prediction probability distribution [27].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Watch Out for the Confusing Faces: Detecting Face Swapping with the Probability Distribution of Face Identification Models

Duan¹,

Zhang²,

Yu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Recently, face swapping has been developing rapidly and achieved a surprising reality, raising concerns about fake content. As a countermeasure, various detection approaches have been proposed and achieved promising performance. However, most existing detectors struggle to maintain performance on unseen face swapping methods and low-quality images. Apart from the generalization problem, current detection approaches have been shown vulnerable to evasion attacks crafted by detection-aware manipulators. Lack of robustness under adversary scenarios leaves threats for applying face swapping detection in real world. In this paper, we propose a novel face swapping detection approach based on face identification probability distributions, coined as IdP FSD, to improve the generalization and robustness. IdP FSD is specially designed for detecting swapped faces whose identities belong to a finite set, which is meaningful in real-world applications. Compared with previous general detection methods, we make use of the available real faces with concerned identities and require no fake samples for training. IdP FSD exploits face swapping's common nature that the identity of swapped face combines that of two faces involved in swapping. We reflect this nature with the confusion of a face identification model and measure the confusion with the maximum value of the output probability distribution. What's more, to defend our detector under adversary scenarios, an attention-based finetuning scheme is proposed for the face identification models used in IdP FSD. Extensive experiments show that the proposed IdP FSD not only achieves high detection performance on different benchmark datasets and image qualities but also raises the bar for manipulators to evade the detection.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Identity Combining Nature Of Face Swappingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Watch Out for the Confusing Faces: Detecting Face Swapping with the Probability Distribution of Face Identification Models

Duan¹,

Zhang²,

Yu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…MegaFS [ZLW*21] shows results in megapixel resolution but has some difficulties in preserving the full source identity due to the pre‐trained StyleGAN2 prior [KLA*20]. In [XDW*22] the authors try to overcome the limitations caused by the pretrained GAN prior through disentangeling the latent semantics and deriving structure and appearance attributes from different decoder layers. [LWXS22] propose an end‐to‐end framework where attributes and identity are disentangled by dedicated encoders.…”

Section: Related Workmentioning

confidence: 99%

Learning Dynamic 3D Geometry and Texture for Video Face Swapping

Otto

Naruniec

Helminger

et al. 2022

Computer Graphics Forum

View full text Add to dashboard Cite

Face swapping is the process of applying a source actor's appearance to a target actor's performance in a video. This is a challenging visual effect that has seen increasing demand in film and television production. Recent work has shown that data‐driven methods based on deep learning can produce compelling effects at production quality in a fraction of the time required for a traditional 3D pipeline. However, the dominant approach operates only on 2D imagery without reference to the underlying facial geometry or texture, resulting in poor generalization under novel viewpoints and little artistic control. Methods that do incorporate geometry rely on pre‐learned facial priors that do not adapt well to particular geometric features of the source and target faces. We approach the problem of face swapping from the perspective of learning simultaneous convolutional facial autoencoders for the source and target identities, using a shared encoder network with identity‐specific decoders. The key novelty in our approach is that each decoder first lifts the latent code into a 3D representation, comprising a dynamic face texture and a deformable 3D face shape, before projecting this 3D face back onto the input image using a differentiable renderer. The coupled autoencoders are trained only on videos of the source and target identities, without requiring 3D supervision. By leveraging the learned 3D geometry and texture, our method achieves face swapping with higher quality than when using off‐the‐shelf monocular 3D face reconstruction, and overall lower FID score than state‐of‐the‐art 2D methods. Furthermore, our 3D representation allows for efficient artistic control over the result, which can be hard to achieve with existing 2D approaches.

show abstract

Information Adversarial Disentanglement for Face Swapping

Chen

et al. 2022

Pattern Recognition and Computer Vision

View full text Add to dashboard Cite

show abstract

High-resolution Face Swapping via Latent Semantics Disentanglement

Cited by 55 publications

References 39 publications

Watch Out for the Confusing Faces: Detecting Face Swapping with the Probability Distribution of Face Identification Models

Watch Out for the Confusing Faces: Detecting Face Swapping with the Probability Distribution of Face Identification Models

Learning Dynamic 3D Geometry and Texture for Video Face Swapping

Information Adversarial Disentanglement for Face Swapping

Contact Info

Product

Resources

About