FaceForensics++: Learning to Detect Manipulated Facial Images

Abstract

Figure 1: FaceForensics++ is a dataset of facial forgeries that enables researchers to train deep-learning-based approaches in a supervised fashion. The dataset contains manipulations created with four state-of-the-art methods, namely, Face2Face, FaceSwap, DeepFakes, and NeuralTextures. AbstractThe rapid progress in synthetic image generation and manipulation has now come to a point where it raises significant concerns for the implications towards society. At best, this leads to a loss of trust in digital content, but could potentially cause further harm by spreading false information or fake news. This paper examines the realism of state-ofthe-art image manipulations, and how difficult it is to detect them, either automatically or by humans.To standardize the evaluation of detection methods, we propose an automated benchmark for facial manipulation detection 1 . In particular, the benchmark is based on Deep-Fakes [1], Face2Face [59], FaceSwap [2] and NeuralTextures [57] as prominent representatives for facial manipulations at random compression level and size. The benchmark is publicly available 2 and contains a hidden test set as well as a database of over 1.8 million manipulated images. This dataset is over an order of magnitude larger than comparable, publicly available, forgery datasets. Based on this data, we performed a thorough analysis of data-driven forgery detectors. We show that the use of additional domainspecific knowledge improves forgery detection to unprecedented accuracy, even in the presence of strong compression, and clearly outperforms human observers.

Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos

“…In our evaluation of the proposed method, we first tested the improvements made to the Capsule-Forensics network: (1) using a larger input size (300 × 300), (2) using dropout on training, and (3) using a larger number of primary capsules (ten). To make the results more convincing, we tested these settings on a large and challenging database -the FaceForensics++ database [27], which focuses on computermanipulated images and videos. After figuring out the best settings, we evaluated its ability to detect fully computergenerated images and presentation attacks.…”

“…To illustrate how Capsule-Forensics works, we used a Capsule-Forensics network with three primary capsules trained on the FaceForensics++ database [27]. We applied both regularizations (using random noise and dropout during training) and used images cropped to 300×300.…”

“…In a test to detect computer-manipulated images and videos, we used the FaceForensics++ database [27], which includes three kinds of manipulations (deepfake [4], Face2Face [5], and FaceSwap [27]) and three multiple compression levels (no compression, light compression, and heavy compression). The database was divided into a training set, a validation set, and a test set, as shown in Table 1.…”

Video tampering localisation using features learned from authentic content

“…Deep learning provides a good tool kit for feature discovery from data; however, it is necessarily data hungry. In fields such as video tampering, a large, labelled and sufficiently varied dataset which encompasses multiple examples from many recent techniques does not yet exist, although [4] and its recent successor [5] show great promise. In fast moving fields, a complete dataset may never exist as, in the time taken to gather and label the data, many more new and improved techniques will be discovered.…”

An AI early warning system to monitor online disinformation, stop violence, and protect elections

“…The program has aided the development of technologies ranging from low-level detectors of image manipulation to high-level analysis programs that can establish the provenance of media content. With respect to the former, software now exists to detect photoshopped (or altered) images (Farid 2016), deepfake videos (Rössler et al 2019), and voice-swapped audio (Agarwal et al 2019). With respect to the latter, novel algorithms can perform sophisticated data mining operations to identify related content and trace the order of its creation.…”