Cross-Forgery Analysis of Vision Transformers and CNNs for Deepfake Image Detection

Abstract

Deepfake Generation Techniques are evolving at a rapid pace, making it possible to create realistic manipulated images and videos and endangering the serenity of modern society. The continual emergence of new and varied techniques brings with it a further problem to be faced, namely the ability of deepfake detection models to update themselves promptly in order to be able to identify manipulations carried out using even the most recent methods. This is an extremely complex problem to solve, as training a model requires large amounts of data, which are difficult to obtain if the deepfake generation method is too recent. Moreover, continuously retraining a network would be unfeasible. In this paper, we ask ourselves if, among the various deep learning techniques, there is one that is able to generalise the concept of deepfake to such an extent that it does not remain tied to one or more specific deepfake generation methods used in the training set. We compared a Vision Transformer with an EfficientNetV2 on a cross-forgery context based on the ForgeryNet dataset. From our experiments, It emerges that EfficientNetV2 has a greater tendency to specialize often obtaining better results on training methods while Vision Transformers exhibit a superior generalization ability that makes them more competent even on images generated with new methodologies.

Comparing Vision Transformers and Convolutional Neural Networks for Image Classification: A Literature Review

“…Table 3 lists all the papers selected for this work, the year of publication and the type of publication. A vision transformer for emphysema classification using CT images 2021 Journal [12] Comparing Vision Transformers and Convolutional Nets for Safety Critical Systems 2022 Conference [13] Convolutional Nets Versus Vision Transformers for Diabetic Foot Ulcer Classification 2022 Conference [14] Convolutional Neural Network (CNN) vs Vision Transformer (ViT) for Digital Holography 2022 Conference [15] Cross-Forgery Analysis of Vision Transformers and CNNs for Deepfake Image Detection 2022 Conference [16] Traffic Sign Recognition with Vision Transformers 2022 Conference [17] An improved transformer network for skin cancer classification 2022 Journal…”

“…The authors in [15] performed an analysis between ViT and CNN models aimed at detecting deepfake images. The experiment consisted in using the ForgeryNet dataset with 2.9 million images and 220 thousand video clips, together with three different image manipulation techniques, where they tried to train the models with real and manipulated images.…”

“…Another study [5] found that ViTs are more robust to adversarial attacks and that CNNs are more sensitive to high-frequency features. The authors in [15] found that CNNs had higher training accuracy and better generalization, but ViTs showed potential to reduce bias in anomaly detection. In [17], the authors claimed that the ViT model showed better performance for skin cancer classification.…”

MAD '22 Workshop: Multimedia AI against Disinformation

“…Their method manages to use a reduced number of parameters while keeping a high accuracy on the predictions. Coccomini et al [3] focus on the challenge of generalizing deepfake detection approaches so that they do not remain tied to one or more specific deepfake generation methods. The audio modality requires different approaches.…”

Ewolucja Cyberzagrożeń: Deepfake i Media Syntetyczne w kontekście bezpieczeństwa energetycznego Europy Wschodniej

“…Media syntetyczne wprowadzają tym samym nową broń do arsenału cyberzagrożeń. W sektorze energetycznym, złożonym i wrażliwym pod względem infrastruktury i ekonomicznej stabilności, tego rodzaju działania dezinformacyjne mogą prowadzić zarówno do zakłóceń operacyjnych, ale także utraty zaufania wśród interesariuszy oraz, w najbardziej ekstremalnych przypadkach, wywołania destabilizującego wpływu na całą gospodarkę i bezpieczeństwo państwa 6 . Niniejszy artykuł ma na celu analizę zagrożeń związanych z wykorzystaniem mediów syntetycznych, ze szczególnym uwzględnieniem wykorzystania mediów syntetycznych takich jak: techniki deepfake, w sektorze energetycznym Europy Wschodniej 7 .…”