UIA-ViT: Unsupervised Inconsistency-Aware Method Based on Vision Transformer for Face Forgery Detection

Abstract

Intra-frame inconsistency has been proved to be effective for the generalization of face forgery detection. However, learning to focus on these inconsistency requires extra pixel-level forged location annotations. Acquiring such annotations is non-trivial. Some existing methods generate large-scale synthesized data with location annotations, which is only composed of real images and cannot capture the properties of forgery regions. Others generate forgery location labels by subtracting paired real and fake images, yet such paired data is difficult to collected and the generated label is usually discontinuous. To overcome these limitations, we propose a novel Unsupervised Inconsistency-Aware method based on Vision Transformer, called UIA-ViT, which only makes use of video-level labels and can learn inconsistency-aware feature without pixel-level annotations. Due to the self-attention mechanism, the attention map among patch embeddings naturally represents the consistency relation, making the vision Transformer suitable for the consistency representation learning. Based on vision Transformer, we propose two key components: Unsupervised Patch Consistency Learning (UPCL) and Progressive Consistency Weighted Assemble (PCWA). UPCL is designed for learning the consistency-related representation with progressive optimized pseudo annotations. PCWA enhances the final classification embedding with previous patch embeddings optimized by UPCL to further improve the detection performance. Extensive experiments demonstrate the effectiveness of the proposed method.

Watch Out for the Confusing Faces: Detecting Face Swapping with the Probability Distribution of Face Identification Models

“…To tackle the malicious use of face swapping, numerous detection methods have been proposed. Most existing works model face swapping detection as a binary classification problem and focus on designing better the features classified on [8], [10], [11], [13], [16], [36], the classifier network [7], [9], [12], [15], [37], [38] or the training policies [14], [39], [40] to improve the accuracy and generalization. For example, low-level artifacts are used in early detection methods, such as face warping artifacts [8].…”

“…As for the network, powerful backbones like Xception [9] and EfficientNet [12] are usually used. Recent works like UIA-VIT [37] utilize Transformer in face forgery detection and achieve high performance. Besides, different training policies such as contrastive learning [39] are applied to make the classifier generalize better on unseen face swapping methods.…”

Real Face Foundation Representation Learning for Generalized Deepfake Detection

“…We adopt the HQ version of FF for both training and testing, and only use one frame every video for testing. We compare our results with state-of-the-art image-based methods Multi-Attention [45], DCL [40], RECCE [3] and UIA-ViT [47]. We ran the public code of RECCE and UIA-ViT to produce results under the same setting.…”

“…We adopt the HQ version of FF for training, and only use one frame every video for testing. Under the same setting, we ran the public code of RECCE [3], UIA-ViT [47] and SBI [38] to produce corresponding results. In Table 2, we show a competitive performance with existing image-based methods, signaling satisfying adaptability of RFFR to different datasets, especially high quality datasets like Celeb-DF.…”

“…A number of studies in recent years devote to performing generalized deepfake detection [3,4,13,17,18,26,35,37,38,40,45,47] where a model is required to identify face images or videos manipulated with methods unseen during training. Though some progress are made, existing results on cross-manipulation tests are still far from satisfying.…”