AIRD: Adversarial Learning Framework for Image Repurposing Detection

Abstract

Image repurposing is a commonly used method for spreading misinformation on social media and online forums, which involves publishing untampered images with modified metadata to create rumors and further propaganda. While manual verification is possible, given vast amounts of verified knowledge available on the internet, the increasing prevalence and ease of this form of semantic manipulation call for the development of robust automatic ways of assessing the semantic integrity of multimedia data. In this paper, we present a novel method for image repurposing detection that is based on the real-world adversarial interplay between a bad actor who repurposes images with counterfeit metadata and a watchdog who verifies the semantic consistency between images and their accompanying metadata, where both players have access to a reference dataset of verified content, which they can use to achieve their goals. The proposed method exhibits state-of-the-art performance on location-identity, subject-identity and painting-artist verification, showing its efficacy across a diverse set of scenarios.

Multimodal news analytics using measures of cross-modal entity and context consistency

“…Also related to our approach are systems for image repurposing detection [21,22,36] that intend to reveal inconsistencies between image-text-pairs with respect to entity representations (persons, locations, organizations, etc. ), mainly to identify repurposed multimedia contents that might indicate misinformation.…”

“…They have also refined the multimodal model using a multitask learning approach that further incorporates geographical information. Jaiswal et al [22] presented an adversarial neural network that simultaneously trains a bad actor who intentionally counterfeits metadata and a watchdog that verifies multimodal semantic consistency. The system was tested for person verification, location verification, and painter verification of artworks.…”

“…While part of previous work [16,25,32,44,46] aims at finding measures to model semantic cross-modal relations in order to bridge the semantic gap, approaches on image repurposing detection [21,22,36] check the consistency of named entities mentioned in the text, as illustrated in Fig. 1.…”

“…Our approach is similar to the task of image repurposing detection since it focuses on the evaluation of cross-modal entity occurrences between image and text. Related approaches [21,22,36] rely on multimodal deep learning techniques that require appropriate datasets of non-manipulated image-text pairs. However, these datasets are hard to collect as they need to be verified for valid cross-modal relations.…”

“…Besides, the training or reference data provide the source of world knowledge and limit these methods to entities, e.g., persons or locations, that appear in these datasets. Experimental evaluations have been performed on images with short image captions [21,36] or existing metadata [22], which do not reflect real-world characteristics as illustrated in Fig. 1.…”