Prompting for Multimodal Hateful Meme Classification

Cao, Rui; Lee, Roy Ka-Wei; Chong, Wen-Haw; Jiang, Jin

doi:10.48550/arxiv.2302.04156

Cited by 2 publications

(3 citation statements)

References 26 publications

(49 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Without any training or fine-tuning, a powerful model like CLIP can produce zero-shot predictions. In order to achieve that, we offer the model some text prompts [27][28]. These text labels or prompts are encoded by the CLIP classifier into a learned latent space, and their similarity to the image latent space is assessed.…”

Section: Prompt Engineeringmentioning

confidence: 99%

Multimodal Hate Speech Detection in Memes Using Contrastive Language-Image Pre-Training

Arya,

Hasan,

Bagwari

et al. 2024

IEEE Access

View full text Add to dashboard Cite

In contemporary society, the proliferation of online hateful messages has emerged as a pressing concern, inflicting deleterious consequences on both societal fabric and individual well-being. The automatic detection of such malevolent content online using models designed to recognize it, holds promise in mitigating its harmful impact. However, the advent of "Hateful Memes" poses fresh challenges to the detection paradigm, particularly within the realm of deep learning models. These memes, constituting of a textual element associated with an image are individually innocuous but their combination causes a detrimental effect. Consequently, entities responsible for disseminating information via web browsers are compelled to institute mechanisms that regulate and automatically filter out such injurious content. Effectively identifying hateful memes demands algorithms and models endowed with robust vision and language fusion capabilities, capable of reasoning across diverse modalities. This research introduces a novel approach by leveraging the multimodal Contrastive Language-Image Pre-Training (CLIP) model, fine-tuned through the incorporation of prompt engineering. This innovative methodology achieves a commendable accuracy of 87.42%. Comprehensive metrics such as loss, AUROC, and f1 score are also meticulously computed, corroborating the efficacy of the proposed strategy. Our findings suggest that this approach presents an efficient means to regulate the dissemination of hate speech in the form of viral meme content across social networking platforms, thereby contributing to a safer online environment.

show abstract

Section: Prompt Engineeringmentioning

confidence: 99%

Multimodal Hate Speech Detection in Memes Using Contrastive Language-Image Pre-Training

Arya,

Hasan,

Bagwari

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Each supervised classification tasks are transformed into a language problems for training. • PromptHate [1]. An encoder model that effectively utilizes pre-trained language models by employing prompts and leveraging implicit knowledge for classifying hateful memes.…”

Section: Modelsmentioning

confidence: 99%

“…These datasets serve as the foundation for training and evaluating models, enabling researchers to create more accurate and robust solutions for classifying memes across different domains. Researchers have explored different approaches, including classic two-stream models that combine text and visual features for classifying hateful memes [4,26], as well as fine-tuning large-scale pre-trained multimodal models for multimodal classification tasks [1,13,16,21,23,29]. However, a challenge arises when models and datasets are scattered across separate GitHub repositories.…”

Section: Introductionmentioning

confidence: 99%

MATK: The Meme Analytical Tool Kit

Hee,

Kumaresan,

Hoang

et al. 2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

The rise of social media platforms has brought about a new digital culture called memes. Memes, which combine visuals and text, can strongly influence public opinions on social and cultural issues. As a result, people have become interested in categorizing memes, leading to the development of various datasets and multimodal models that show promising results in this field. However, there is currently a lack of a single library that allows for the reproduction, evaluation, and comparison of these models using fair benchmarks and settings. To fill this gap, we introduce the Meme Analytical Tool Kit (MATK), an open-source toolkit specifically designed to support existing memes datasets and cutting-edge multimodal models. MATK aims to assist researchers and engineers in training and reproducing these multimodal models for meme classification tasks, while also providing analysis techniques to gain insights into their strengths and weaknesses. To access MATK, please visit https://github.com/Social-AI-Studio/MATK. CCS CONCEPTS• Computing methodologies → Natural language processing; Computer vision representations.

show abstract

Prompting for Multimodal Hateful Meme Classification

Cited by 2 publications

References 26 publications

Multimodal Hate Speech Detection in Memes Using Contrastive Language-Image Pre-Training

Multimodal Hate Speech Detection in Memes Using Contrastive Language-Image Pre-Training

MATK: The Meme Analytical Tool Kit

Contact Info

Product

Resources

About