Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos

Abstract

The revolution in computer hardware, especially in graphics processing units and tensor processing units, has enabled significant advances in computer graphics and artificial intelligence algorithms. In addition to their many beneficial applications in daily life and business, computergenerated/manipulated images and videos can be used for malicious purposes that violate security systems, privacy, and social trust. The deepfake phenomenon and its variations enable a normal user to use his or her personal computer to easily create fake videos of anybody from a short real online video. Several countermeasures have been introduced to deal with attacks using such videos. However, most of them are targeted at certain domains and are ineffective when applied to other domains or new attacks. In this paper, we introduce a capsule network that can detect various kinds of attacks, from presentation attacks using printed images and replayed videos to attacks using fake videos created using deep learning. It uses many fewer parameters than traditional convolutional neural networks with similar performance. Moreover, we explain, for the first time ever in the literature, the theory behind the application of capsule networks to the forensics problem through detailed analysis and visualization.

Learn with diversity and from harder samples: Improving the generalization of CNN-Based detection of computer-generated images

“…For the CG image forensic problem, there mainly exist two types of methods: hand-crafted-feature-based methods [6,7,8,9,10,11,12] and CNN-based methods [13,14,15,16,17,18,19].…”

“…Inspired by the notable success of CNN in the field of computer vision and pattern recognition, some recent works also applied CNN to solve the CG image forensic problem [13,14,15,16,17,18,19]. Rahmouni et al [13] explicitly extracted low-order statistical information of convoluted image as discriminative features and trained a model to distinguish computer graphics from photographic images.…”

“…He et al [16] combined CNN and recurrent neural network (RNN) [20] to detect CG images. Nguyen et al [17] used the capsule network [21] for the CG image forensic problem. Bhalang Tarianga et al [18] proposed an attention-based recurrent model to classify computer-generated and natural images.…”

“…In the past decades, researchers have proposed hand-crafted-feature-based methods [6,7,8,9,10,11,12] and CNN(convolutional neural network)-based methods [13,14,15,16,17,18,19] to solve this forensic problem. The former usually consists of multidimensional feature extraction and classifier training, and the latter directly formulates an optimization problem to find a good mapping function from a given image to its label and solves it efficiently in an end-to-end manner.…”

Classification of LiDAR Data Combined Octave Convolution With Capsule Network

“…Anupama et al [26] introduced CapsNet into the breast cancer classification task and obtained the best results compared to different CNN architectures [26]. Nguyen et al [27] combined VGG with CapsNet to detect various kinds of forged images and videos [27]. Gumusbas and Yildirim [28] used CapsNet in the signature identification task [28].…”