Characterizing the Entities in Harmful Memes: Who is the Hero, the Villain, the Victim?

Abstract

Memes can sway people's opinions over social media as they combine visual and textual information in an easy-to-consume manner. Since memes instantly turn viral, it becomes crucial to infer their intent and potentially associated harmfulness to take timely measures as needed. A common problem associated with meme comprehension lies in detecting the entities referenced and characterizing the role of each of these entities. Here, we aim to understand whether the meme glorifies, vilifies, or victimizes each entity it refers to. To this end, we address the task of role identification of entities in harmful memes, i.e., detecting who is the 'hero', the 'villain', and the 'victim' in the meme, if any. We utilize HVVMemes -a memes dataset on US Politics and Covid-19 memes, released recently as part of the CONSTRAINT@ACL-2022 shared-task. It contains memes, entities referenced, and their associated roles: hero, villain, victim, and other. We further design VECTOR (Visual-semantic role dEteC-ToR), a robust multi-modal framework for the task, which integrates entity-based contextual information in the multi-modal representation and compare it to several standard unimodal (text-only or image-only) or multi-modal (im-age+text) models. Our experimental results show that our proposed model achieves an improvement of 4% over the best baseline and 1% over the best competing stand-alone submission from the shared-task. Besides divulging an extensive experimental setup with comparative analyses, we finally highlight the challenges encountered in addressing the complex task of semantic role labeling within memes.

Capturing the Concept Projection in Metaphorical Memes for Downstream Learning Tasks

“…On the other hand, the main external knowledge sources are Word-Net, Wikidata, DBPedia, ConceptNet [61], and Visual Genome [62]. The recent work on multimodal memes has mainly used ConceptNet [16], [17], [63]- [65] as an external knowledge source. The reason for choosing ConceptNet is that it encompasses several knowledge categories, such as part-whole relationships, utility relationships, factual knowledge, behavioral knowledge, common event knowledge, etc., which otherwise have to be sourced from individual knowledge repositories.…”

“…• Strategies based on Attention and Fusion: Shivam Sharma et al, in their AOMD framework [15], used comments on social media as the source of external knowledge along with the Cross-Attention mechanism in the model for detection of offensive memes. Shivam Sharma et al, in a recent work [17], have combined information from images, text, and knowledge from ConceptNet. Although a Graph neural network has been used to get a knowledge graph embedding of acquired knowledge from ConceptNet, the main concept transitioning mechanism is a state-of-the-art Optimal Transport Based Kernel Embedding Layer [80] replacing the traditional cross-attention mechanism.…”

“…Compared to NLU and Computer Vision, more work must be done on metaphorical memes and consequent downstream learning tasks. Existing research has been primarily about downstream discriminative tasks that do not use metaphor detection as a building block, i.e., detection of hateful memes [14] and offensive memes [15], sarcasm detection [16], characterization of meme entities [17], and detection of sentiment [18]- [21] as well as emotion in memes. Even for purely metaphor-centric research on memes, there needs to be more work in metaphor identification and interpretation.…”