Combining EfficientNet and Vision Transformers for Video Deepfake Detection

Coccomini, Davide Alessandro; Messina, Nicola; Gennaro, Claudio; Falchi, Fabrizio

doi:10.1007/978-3-031-06433-3_19

Cited by 120 publications

(50 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, if the blocks of the image are shuffled in a proper way, it can lead to preserving essential characteristics while also enhancing the quality of the model [55], [56]. Additionally, several researches in [57] and [58] have demonstrated that creating patches by using characteristics gathered in an image also increases the quality of the training process. Therefore, these demonstrate that block shifting and shuffling local regions greatly raise quality when applied properly.…”

Section: Data Preprocessingmentioning

confidence: 99%

Generalization of Forgery Detection With Meta Deepfake Detection Model

et al. 2023

View full text Add to dashboard Cite

Face forgery generating algorithms that produce a range of manipulated videos/images have developed quickly. Consequently, this causes an increase in the production of fake information, making it difficult to identify. Because facial manipulation technologies raise severe concerns, face forgery detection is gaining increasing attention in the area of computer vision. In real-world applications, face forgery detection systems frequently encounter and perform poorly in unseen domains, due to poor generalization. In this paper, we propose a deepfake detection method based on meta-learning called Meta Deepfake Detection (MDD). The goal of the model is to develop a generalized model capable of directly solving new unseen domains without the need for model updates. The MDD algorithm establishes various weights for facial images from various domains. Specifically, MDD uses meta-weight learning to shift information from the source domains to the target domains with meta-optimization steps, which aims for the model to generate effective representations of the source and target domains. We build multi-domain sets using meta splitting strategy to create a meta-train set and meta-test set. Based on these sets, the model determines the gradient descent and obtains backpropagation. The inner and outer loop gradients were aggregated to update the model to enhance generalization. By introducing pair-attention loss and average-center alignment loss, the detection capabilities of the system were substantially enhanced. In addition, we used some evaluation benchmarks established from several popular deepfake datasets to compare the generalization of our proposal in several baselines and assess its effectiveness.

show abstract

Section: Data Preprocessingmentioning

confidence: 99%

Generalization of Forgery Detection With Meta Deepfake Detection Model

et al. 2023

View full text Add to dashboard Cite

show abstract

“…The further the generations go, the larger these datasets are, and the more frames they contain. Recently, with the increased focus on the concept of identifying Deepfakes ’in the wild’, a number of further important datasets have emerged, namely OpenForensics [ 26 ] which seeks to provide images containing multiple faces or crowds of people to address the problem of multi-face forgery detection as also reported in [ 27 ] and WildDeepfake [ 28 ], which aims to provide a wide variety of scenarios, situations, techniques and perturbations in the images and videos within them. Finally, a particularly large and varied dataset was presented, namely ForgeryNet [ 29 ], containing millions of images and hundreds of thousands of videos crafted with dozens of manipulation techniques, perturbations and a great variety of scenes and identities.…”

Section: Deepfake Literature Overviewmentioning

confidence: 99%

“…As the images provided in the challenge dataset are already facial cutouts of the people, this last step was not applied for this specific dataset. Inspired by previous work [ 27 ], participants proposed a mixed architecture between a Cross Vision Transformer [ 52 ] and an EfficientNets [ 53 , 54 ]. When working with Vision Transformers, the first step is to split the input image into several non-overlapping patches of equal size.…”

Section: Researcher Solutionsmentioning

confidence: 99%

“…In this case, we chose to use some networks from the EfficientNet category as convolutional backbones. The choice of these specific convolutional backbones stems from the observation that the EfficientNet category seems to have been particularly effective on the Deepfake Detection task in previous works [ 27 , 30 , 33 ]. Other research work has, however, used larger versions of the EfficientNet, such as the EfficientNet-B7, but since the aim in this case was also to obtain an architecture that was as light as possible, we opted for lighter alternatives for the experiments, namely the EfficientNet-B0 and the EfficientNetV2-M.…”

Section: Researcher Solutionsmentioning

confidence: 99%

See 1 more Smart Citation

The Face Deepfake Detection Challenge

et al. 2022

View full text Add to dashboard Cite

Multimedia data manipulation and forgery has never been easier than today, thanks to the power of Artificial Intelligence (AI). AI-generated fake content, commonly called Deepfakes, have been raising new issues and concerns, but also new challenges for the research community. The Deepfake detection task has become widely addressed, but unfortunately, approaches in the literature suffer from generalization issues. In this paper, the Face Deepfake Detection and Reconstruction Challenge is described. Two different tasks were proposed to the participants: (i) creating a Deepfake detector capable of working in an “in the wild” scenario; (ii) creating a method capable of reconstructing original images from Deepfakes. Real images from CelebA and FFHQ and Deepfake images created by StarGAN, StarGAN-v2, StyleGAN, StyleGAN2, AttGAN and GDWCT were collected for the competition. The winning teams were chosen with respect to the highest classification accuracy value (Task I) and “minimum average distance to Manhattan” (Task II). Deep Learning algorithms, particularly those based on the EfficientNet architecture, achieved the best results in Task I. No winners were proclaimed for Task II. A detailed discussion of teams’ proposed methods with corresponding ranking is presented in this paper.

show abstract

“…Aside from using cryptographic primitives, the literature has proposed detecting deepfakes to combat them [23]. There are a range of benchmarks [9,41] and techniques proposed to detect deepfakes [7,16,40]. While promising, there are a number of challenges with these methods.…”

Section: Related Workmentioning

confidence: 99%

ZK-IMG: Attested Images via Zero-Knowledge Proofs to Fight Disinformation

Kang¹,

Hashimoto²,

Stoica³

et al. 2022

Preprint

View full text Add to dashboard Cite

Over the past few years, AI methods of generating images have been increasing in capabilities, with recent breakthroughs enabling high-resolution, photorealistic "deepfakes" (artificially generated images with the purpose of misinformation or harm). The rise of deepfakes has potential for social disruption. Recent work has proposed using ZK-SNARKs (zero-knowledge succinct non-interactive argument of knowledge) and attested cameras to verify that images were taken by a camera. ZK-SNARKs allow verification of image transformations non-interactively (i.e., post-hoc) with only standard cryptographic hardness assumptions. Unfortunately, this work does not preserve input privacy, is impractically slow (working only on 128×128 images), and/or requires custom cryptographic arguments.To address these issues, we present ZK-IMG, a library for attesting to image transformations while hiding the pretransformed image. ZK-IMG allows application developers to specify high level image transformations. Then, ZK-IMG will transparently compile these specifications to ZK-SNARKs. To hide the input or output images, ZK-IMG will compute the hash of the images inside the ZK-SNARK. We further propose methods of chaining image transformations securely and privately, which allows for arbitrarily many transformations. By combining these optimizations, ZK-IMG is the first system to be able to transform HD images on commodity hardware, securely and privately.

show abstract

Combining EfficientNet and Vision Transformers for Video Deepfake Detection

Cited by 120 publications

References 25 publications

Generalization of Forgery Detection With Meta Deepfake Detection Model

Generalization of Forgery Detection With Meta Deepfake Detection Model

The Face Deepfake Detection Challenge

ZK-IMG: Attested Images via Zero-Knowledge Proofs to Fight Disinformation

Contact Info

Product

Resources

About