Detect and Locate: Exposing Face Manipulation by Semantic- and Noise-Level Telltales

Kong, Chenqi; Chen, Baoliang; Li, Haoliang; Wang, Shiqi; Rocha, Anderson

doi:10.1109/tifs.2022.3169921

Cited by 47 publications

(12 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To tackle the malicious use of face swapping, numerous detection methods have been proposed. Most existing works model face swapping detection as a binary classification problem and focus on designing better the features classified on [8], [10], [11], [13], [16], [36], the classifier network [7], [9], [12], [15], [37], [38] or the training policies [14], [39], [40] to improve the accuracy and generalization. For example, low-level artifacts are used in early detection methods, such as face warping artifacts [8].…”

Section: Related Workmentioning

confidence: 99%

“…To tackle the malicious application of face swapping, various detection methods have been proposed. A common way is to model face swapping detection as a binary classification task and collect existing face forgery images to train the classifier [7], [8], [9], [10], [11], [12], [13], [14], [15], [16]. Although these approaches achieve high detection accuracy on benchmark datasets, they still struggle to meet some real-world demands.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Watch Out for the Confusing Faces: Detecting Face Swapping with the Probability Distribution of Face Identification Models

Duan¹,

Zhang²,

Yu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Recently, face swapping has been developing rapidly and achieved a surprising reality, raising concerns about fake content. As a countermeasure, various detection approaches have been proposed and achieved promising performance. However, most existing detectors struggle to maintain performance on unseen face swapping methods and low-quality images. Apart from the generalization problem, current detection approaches have been shown vulnerable to evasion attacks crafted by detection-aware manipulators. Lack of robustness under adversary scenarios leaves threats for applying face swapping detection in real world. In this paper, we propose a novel face swapping detection approach based on face identification probability distributions, coined as IdP FSD, to improve the generalization and robustness. IdP FSD is specially designed for detecting swapped faces whose identities belong to a finite set, which is meaningful in real-world applications. Compared with previous general detection methods, we make use of the available real faces with concerned identities and require no fake samples for training. IdP FSD exploits face swapping's common nature that the identity of swapped face combines that of two faces involved in swapping. We reflect this nature with the confusion of a face identification model and measure the confusion with the maximum value of the output probability distribution. What's more, to defend our detector under adversary scenarios, an attention-based finetuning scheme is proposed for the face identification models used in IdP FSD. Extensive experiments show that the proposed IdP FSD not only achieves high detection performance on different benchmark datasets and image qualities but also raises the bar for manipulators to evade the detection.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Watch Out for the Confusing Faces: Detecting Face Swapping with the Probability Distribution of Face Identification Models

Duan¹,

Zhang²,

Yu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Early forged face videos tend to exhibit specific artifacts [37] in both spatial and temporal domains, which inspires some works to exploit hand-crafted features for forgery detection, such as lack of eye blinking [8], inconsistency of head pose [9], and heart rate artifacts [38]. Powerful as deep models are, recent works take the Xception [39] and EfficientNet [31] as backbone and incorporate various types of prior knowledge to boost the detection performance [17], [22], [26], [40]- [43]. For instance, Face X-ray [23] learns to identify the boundary inconsistency left by the blending operation in forged images, while SBI [24] enhances the blending artifact by synthesizing more challenging fake samples in a self-blending manner.…”

Section: A Prior Knowledge-based Face Forgery Detectionmentioning

confidence: 99%

“…In [16]- [19], various subtle and finegrained clues have been identified as significant evidence for face forgery detection. The experts' prior forgery knowledge, including noise patterns [20]- [22], boundary artifacts [23], [24], and frequency information [25], [26], have been widely studied to improve the generalization capability. However, these specific clues can be easily targeted by an expert attacker and are not robust to various image distortions.…”

Section: Introductionmentioning

confidence: 99%

Dual Attention Network Approaches to Face Forgery Video Detection

Luo

Chen

2022

IEEE Access

View full text Add to dashboard Cite

Forged videos are commonly spread online. Most have malicious content and cause serious information security problems. The most critical issue in deepfake detection is the identification of traces of tampering in fake videos. This study designs a Dual Attention Forgery Detection Network (DAFDN), which embeds a spatial reduction attention block (SRAB) and a forgery feature attention module (FFAM) to the backbone network. DAFDN embeds the two proposed attention mechanisms and enables the convolution neural network to extract peculiar traces left by images' warping. This study uses two benchmark datasets, DFDC and FaceForensics++, to compare the performance of the proposed DAFDN with other methods. The results show that the proposed DAFDN mechanism achieves AUC scores of 0.911 and 0.945 in the datasets DFDC and FaceForensics++, respectively. These results are better than those of previously developed methods, such as XceptionNet and EfficientNet-related methods.

show abstract

“…However, these methods suffer from limited accuracy and low generalization capability. Thanks to the advent of artificial intelligence and deep learning, many learning-based detection methods [34,32,10,26,18,30] have been proposed and achieved outstanding detection performance under intra-domain settings. Nonetheless, learning-based methods are prone to overfitting to the training data, resulting in dramatic performance drops when deployed to unforeseen domains.…”

Section: Introductionmentioning

confidence: 99%

Face Forgery Detection via Symmetric Transformer

Song

Fang³

et al. 2022

Proceedings of the 30th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Nowadays, forgery faces pose pressing security concerns over fake news, fraud, impersonation, etc. Despite the demonstrated success in intra-domain face forgery detection, existing detection methods lack generalization capability and tend to suffer from dramatic performance drops when deployed to unforeseen domains. To mitigate this issue, this paper designs a more general fake face detection model based on the vision transformer(ViT) architecture. In the training phase, the pretrained ViT weights are freezed, and only the Low-Rank Adaptation(LoRA) modules are updated. Additionally, the Single Center Loss(SCL) is applied to supervise the training process, further improving the generalization capability of the model. The proposed method achieves state-of-the-arts detection performances in both cross-manipulation and cross-dataset evaluations.

show abstract

Detect and Locate: Exposing Face Manipulation by Semantic- and Noise-Level Telltales

Cited by 47 publications

References 51 publications

Watch Out for the Confusing Faces: Detecting Face Swapping with the Probability Distribution of Face Identification Models

Watch Out for the Confusing Faces: Detecting Face Swapping with the Probability Distribution of Face Identification Models

Dual Attention Network Approaches to Face Forgery Video Detection

Face Forgery Detection via Symmetric Transformer

Contact Info

Product

Resources

About