Video Transformer for Deepfake Detection with Incremental Learning

Khan, Sohail Ahmed; Dai, Hang

doi:10.1145/3474085.3475332

Cited by 42 publications

(15 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The community lacks an end-to-end vision-transformer framework designed explicitly for deepfake detection tasks. For example, Khan and Dai [46] presented a video transformer with an incremental learning approach for deepfake detection. Their design benefits from XceptionNet [47] as a backbone for image feature extraction and 12 transformer blocks for feature learning.…”

Section: Related Workmentioning

confidence: 99%

“…This figure also reveals another critical point: although most deepfake detection approaches perform well on relatively more straightforward datasets, i.e., FaceForensics++, their performance is still far from perfect on more challenging and real-world datasets, i.e., WildDeepfake. [46] 99.64 F3-Net [73] 65.1 ADD-Xception [18] 79.23 RNN [74] 83.10 PPA [75] 83.1 DefakeHop [6] 90.5 FakeCatcher [15] 91.5 ATS-DE [7] 97.8 ADD-ResNet [18] 98. MesoNet-4 [34] ADDNet-3D [75] MesoNet-inception [34] XceptionNet [65] ADDNet-2D [75] ADD-Xception [55] DFDT (Ours) ACC (%)…”

Section: Intra-dataset Evaluationmentioning

confidence: 99%

See 1 more Smart Citation

DFDT: An End-to-End DeepFake Detection Framework Using Vision Transformer

Khormali

Yuan

2022

Applied Sciences

View full text Add to dashboard Cite

The ever-growing threat of deepfakes and large-scale societal implications has propelled the development of deepfake forensics to ascertain the trustworthiness of digital media. A common theme of existing detection methods is using Convolutional Neural Networks (CNNs) as a backbone. While CNNs have demonstrated decent performance on learning local discriminative information, they fail to learn relative spatial features and lose important information due to constrained receptive fields. Motivated by the aforementioned challenges, this work presents DFDT, an end-to-end deepfake detection framework that leverages the unique characteristics of transformer models, for learning hidden traces of perturbations from both local image features and global relationship of pixels at different forgery scales. DFDT is specifically designed for deepfake detection tasks consisting of four main components: patch extraction & embedding, multi-stream transformer block, attention-based patch selection followed by a multi-scale classifier. DFDT’s transformer layer benefits from a re-attention mechanism instead of a traditional multi-head self-attention layer. To evaluate the performance of DFDT, a comprehensive set of experiments are conducted on several deepfake forensics benchmarks. Obtained results demonstrated the surpassing detection rate of DFDT, achieving 99.41%, 99.31%, and 81.35% on FaceForensics++, Celeb-DF (V2), and WildDeepfake, respectively. Moreover, DFDT’s excellent cross-dataset & cross-manipulation generalization provides additional strong evidence on its effectiveness.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Intra-dataset Evaluationmentioning

confidence: 99%

DFDT: An End-to-End DeepFake Detection Framework Using Vision Transformer

Khormali

Yuan

2022

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…Deepfake detection is a challenging task and a lot of studies have been proposed in the past to detect deepfake media employing diverse set of features for training deep learning models. Some examples include, biological signals, behavioural features, 3D face decomposition features, and optical flow [117], [118], [119], [120], [121], [122], [123], [109], birds [110], buses [109], indoor and outdoor scenes [109].…”

Section: Deepfake Detectionmentioning

confidence: 99%

Visual User-Generated Content Verification in Journalism: An Overview

et al. 2023

Self Cite

View full text Add to dashboard Cite

Over the past few years, social media has become an indispensable part of the news generation and dissemination cycle on the global stage. These digital channels along with the easy-to-use editing tools have unfortunately created a medium for spreading mis-/disinformation containing visual content. Media practitioners and fact-checkers continue to struggle with scrutinising and debunking visual user-generated content (UGC) quickly and thoroughly as verification of visual content requires a high level of expertise and could be exceedingly complex amid the existing computational tools employed in newsrooms. The aim of this study is to present a forward-looking perspective on how visual UGC verification in journalism can be transformed by multimedia forensics research. We elaborate on a comprehensive overview of the five elements of the UGC verification and propose multimedia forensics as the sixth element. In addition, different types of visual content forgeries and detection approaches proposed by the computer science research community are explained. Finally, a mapping of the available verification tools media practitioners rely on is created along with their limitations and future research directions to gain the confidence of media professionals in using multimedia forensics tools in their day-to-day routine.

show abstract

“…Several works leveraging transformers to boost face forgery detection have been done: Miao et al [18] extend transformer using bag-of-feature to learn local forgery features. Khan et al [8] propose a video transformer to extract spatial features with the temporal information for detecting forgery. Zheng et al [35] design a light-weight temporal transformer after their proposed fully temporal convolution network to explore the temporal coherence for general manipulated video detection.…”

Section: Transformermentioning

confidence: 99%

UIA-ViT: Unsupervised Inconsistency-Aware Method Based on Vision Transformer for Face Forgery Detection

Zhuang

Chu

Tan

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Intra-frame inconsistency has been proved to be effective for the generalization of face forgery detection. However, learning to focus on these inconsistency requires extra pixel-level forged location annotations. Acquiring such annotations is non-trivial. Some existing methods generate large-scale synthesized data with location annotations, which is only composed of real images and cannot capture the properties of forgery regions. Others generate forgery location labels by subtracting paired real and fake images, yet such paired data is difficult to collected and the generated label is usually discontinuous. To overcome these limitations, we propose a novel Unsupervised Inconsistency-Aware method based on Vision Transformer, called UIA-ViT, which only makes use of video-level labels and can learn inconsistency-aware feature without pixel-level annotations. Due to the self-attention mechanism, the attention map among patch embeddings naturally represents the consistency relation, making the vision Transformer suitable for the consistency representation learning. Based on vision Transformer, we propose two key components: Unsupervised Patch Consistency Learning (UPCL) and Progressive Consistency Weighted Assemble (PCWA). UPCL is designed for learning the consistency-related representation with progressive optimized pseudo annotations. PCWA enhances the final classification embedding with previous patch embeddings optimized by UPCL to further improve the detection performance. Extensive experiments demonstrate the effectiveness of the proposed method.

show abstract

Video Transformer for Deepfake Detection with Incremental Learning

Cited by 42 publications

References 18 publications

DFDT: An End-to-End DeepFake Detection Framework Using Vision Transformer

DFDT: An End-to-End DeepFake Detection Framework Using Vision Transformer

Visual User-Generated Content Verification in Journalism: An Overview

UIA-ViT: Unsupervised Inconsistency-Aware Method Based on Vision Transformer for Face Forgery Detection

Contact Info

Product

Resources

About