User guide for NIST Media Forensic Challenge (MFC) datasets

Abstract

More than 300 individuals from 150 organizations across 26 countries and regions use the NIST released Media Forensic Challenge (MFC) datasets for their research. The MFC datasets were created for use in the DARPA MediFor (Media Forensics) program. Since their release, multiple questions have been fielded regarding the dataset properties, including contents, metadata definitions, usage, data repurposing, etc. For example: what do the datasets contain? What are the definitions of the different kinds of metadata? How does one label the data with the reference information to build the training data for machine learning algorithms? How would one modify/extract the data for their own research purposes? This document serves as a user guide for the MFC datasets, including those used in the Nimble Challenge (NC). This guide includes: 1) a description about MFC datasets including background, evolution history, and the dataset summary by the evaluation tasks; 2) user access and permissions of MFC datasets; 3) an introduction to the MFC data by providing a simple example of a manipulation journal graph and its detailed corresponding MFC dataset reference files; 4) an introduction to a flexible subset selection approach, "Selective Scoring," to sample the test probes from the entire test set for the particular task evaluation; 5) information to help users gain a deeper understanding of the metadata by presenting two commonly used approaches to illustrate the manipulation operation statistic histogram distributions, and 6) a general template of the NIST MFC evaluation dataset to facilitate the future dataset generation.

Open Media Forensics Challenge (OpenMFC) 2020-2021 :

“…We designed evaluation tasks, datasets, metrics and the scoring software, and held 4 yearly evaluations to evaluate the DARPA performer teams' systems. We reported the state-of-the-art of media forensic technologies [6]- [21] to fulfill the DARPA mission.…”

“…However, with the emerging and dynamically updated manipulation approaches, the performance of a media forensic system could dramatically decrease, and continuous development and evaluation are essential. The deployment of measurement techniques on media forensics additionally faces the challenging problems of constantly and swiftly adapting datasets [6] [7] [8], evaluation tasks [9], and evaluation metrics [15] [16], which are needed to stay up to date with rapid advancements in AI technologies. Please refer to our previous publications [6] - [21] for the challenges and solutions on designing and developing an evaluation program.…”

“…Since 2016, NIST has released MFC dataset to more than 500 individuals and 200 organizations from 26 countries and regions worldwide. • Published the OpenMFC 2020-2021 evaluation plan [15] and MFC dataset user guide NISTIR report [6]. • Publishing the evaluation report on the current media forensic state-of-the-art in public domain and OpenMFC 2020-2021 result (this report).…”

“…The AUC is 0.99. Compared with the NC16, NC17, and MFC18 datasets, the MFC19 dataset is larger and describes manipulation operations with high quality of annotation (see [6] for the MFC datasets summary). It shows that with the larger data size and more sample data coverage (e.g., the larger number of the manipulation operations), the system performance could improve greatly.…”

“…• Design evaluation tasks varying in difficulty level and technical scope: adapt aforementioned needs from public researchers with different skill sets, we introduce Image Splice Detection (ISD) task to greatly reduces researchers' workload. We design Steganography Detection task [6,7] and Deepfakes Detection task [6,8] focusing on the new advanced emerging topics.…”