On the Generalization of Deep Learning Models in Video Deepfake Detection

Abstract

The increasing use of deep learning techniques to manipulate images and videos, commonly referred to as “deepfakes”, is making it more challenging to differentiate between real and fake content, while various deepfake detection systems have been developed, they often struggle to detect deepfakes in real-world situations. In particular, these methods are often unable to effectively distinguish images or videos when these are modified using novel techniques which have not been used in the training set. In this study, we carry out an analysis of different deep learning architectures in an attempt to understand which is more capable of better generalizing the concept of deepfake. According to our results, it appears that Convolutional Neural Networks (CNNs) seem to be more capable of storing specific anomalies and thus excel in cases of datasets with a limited number of elements and manipulation methodologies. The Vision Transformer, conversely, is more effective when trained with more varied datasets, achieving more outstanding generalization capabilities than the other methods analysed. Finally, the Swin Transformer appears to be a good alternative for using an attention-based method in a more limited data regime and performs very well in cross-dataset scenarios. All the analysed architectures seem to have a different way to look at deepfakes, but since in a real-world environment the generalization capability is essential, based on the experiments carried out, the attention-based architectures seem to provide superior performances.

Harmonizing Image Forgery Detection & Localization: Fusion of Complementary Approaches

“…Future work may tackle the generalization issue by utilizing more training data, from a larger variety of datasets with a larger variety of forgery types. Additionally, the generator architecture in our proposed framework could be replaced with cutting-edge alternatives such as vision transformers [45] in order to further refine the performance, which can be inspired by related work that evaluated the generalization of deep-learning models [28]. Finally, we can improve the overall fusion performance by including new high-performing methods, such as TruFor [46] and FOCAL [47], which are relatively easy to include in our proposed fusion framework.…”

“…These fusion methods make it more practical to add new existing detection methods to the set of input methods. However, a limitation of these models is their lack of generalization to manipulations unseen during training, which is a common problem in supervised learning [28]. Consequently, it occasionally yields worse results compared to the best individual input detection algorithm.…”