An Interpretable Approach to Hateful Meme Detection

Deshpande, Tanvi; Mani, Nitya

doi:10.1145/3462244.3479949

Cited by 10 publications

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The detection of Hateful Memes represents a binary classification challenge, seeking to ascertain whether a meme is offensive or hateful through the analysis of multimodal data constituting both text and image signals. Given the inherent complexity of memes, characterized by the coexistence of two modalities, and recognizing the formidable obstacle of detection accuracy in this task, a multi-task learning technique is deemed essential for drawing meaningful statistical conclusions [10][11].…”

Section: Methodsmentioning

confidence: 99%

Multimodal Hate Speech Detection in Memes Using Contrastive Language-Image Pre-Training

Arya,

Hasan,

Bagwari

et al. 2024

IEEE Access

View full text Add to dashboard Cite

In contemporary society, the proliferation of online hateful messages has emerged as a pressing concern, inflicting deleterious consequences on both societal fabric and individual well-being. The automatic detection of such malevolent content online using models designed to recognize it, holds promise in mitigating its harmful impact. However, the advent of "Hateful Memes" poses fresh challenges to the detection paradigm, particularly within the realm of deep learning models. These memes, constituting of a textual element associated with an image are individually innocuous but their combination causes a detrimental effect. Consequently, entities responsible for disseminating information via web browsers are compelled to institute mechanisms that regulate and automatically filter out such injurious content. Effectively identifying hateful memes demands algorithms and models endowed with robust vision and language fusion capabilities, capable of reasoning across diverse modalities. This research introduces a novel approach by leveraging the multimodal Contrastive Language-Image Pre-Training (CLIP) model, fine-tuned through the incorporation of prompt engineering. This innovative methodology achieves a commendable accuracy of 87.42%. Comprehensive metrics such as loss, AUROC, and f1 score are also meticulously computed, corroborating the efficacy of the proposed strategy. Our findings suggest that this approach presents an efficient means to regulate the dissemination of hate speech in the form of viral meme content across social networking platforms, thereby contributing to a safer online environment.

show abstract

Section: Methodsmentioning

confidence: 99%

Multimodal Hate Speech Detection in Memes Using Contrastive Language-Image Pre-Training

Arya,

Hasan,

Bagwari

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Of 8 papers in the geoscience domain, 5 were in classification [347]- [351], 1 was in recognition [352], detection [353], and precision [354]. In total 10 articles were identified in the social media domain, where 4 were in detection [355]- [358], 2 were in analysis [359], [360] and 1 was in classification [361], prediction [362], recommendation [363] and verification [364]. In the vehicle domain, 6 articles were found; from them, 2 were in recognition [365], [366], 1 was in identification [367] and 3 were in detection [368]- [370].…”

Section: Inclusion Criteriamentioning

confidence: 99%

“…The remaining 57 articles were related to combined models of model-based and agnostic-based models. Seven types of the combination were found: early & late [65], [78], [80], [86], [87], [109], [118], [123], [131], [135], [151], [165], [190], [206], [234], [235], [239], [251], [255], [269], [278], [281], [297], [302], [304], [314], [350], [358], [361], [363], [403], [412], [414], early & late & hybrid [74], [117], [232], [260], [263], [284], [331], [354], [370], early & late & kernel [150], [238], early & late & neural networks [60], [67], [183], [288], [409], early & neural networks [81], [226], [375], late & hybrid [309]<...…”

Section: F Fusionmentioning

confidence: 99%

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

2023

View full text Add to dashboard Cite

Multimodal machine learning (MML) is a tempting multidisciplinary research area where heterogeneous data from multiple modalities and machine learning (ML) are combined to solve critical problems. Usually, research works use data from a single modality, such as images, audio, text, and signals. However, real-world issues have become critical now, and handling them using multiple modalities of data instead of a single modality can significantly impact finding solutions. ML algorithms play an essential role by tuning parameters in developing MML models. This paper reviews recent advancements in the challenges of MML, namely: representation, translation, alignment, fusion and co-learning, and presents the gaps and challenges. A systematic literature review (SLR) applied to define the progress and trends on those challenges in the MML domain. In total, 1032 articles were examined in this review to extract features like source, domain, application, modality, etc. This research article will help researchers understand the constant state of MML and navigate the selection of future research directions.

show abstract

“…Each has its own peculiarities, often related to information extraction techniques, especially from the visual content. Among them, the most used are text removal, object detection, image captioning, and entity recognition, such as in (Deshpande and Mani, 2021), (Pramanick et al, 2021) and (Lee et al, 2021). First of all, we exploit single modalities separately, through state-of-the-art deep models, to investigate their relevance and the quantity of information they carry on.…”

Section: Related Workmentioning

confidence: 99%

JRLV at SemEval-2022 Task 5: The Importance of Visual Elements for Misogyny Identification in Memes

Ravagli¹,

Vaiani²

2022

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)

View full text Add to dashboard Cite

Gender discrimination is a serious and widespread problem on social media and online in general. Besides offensive messages, memes are one of the main means of dissemination for such content. With these premises, the MAMI task was proposed at the SemEval-2022, which consists of identifying memes with misogynous characteristics. In this work, we propose a solution to this problem based on Mask R-CNN and VisualBERT that leverages the multimodal nature of the task. Our study focuses on observing how the two sources of data in memes (text and image) and their possible combinations impact performances. Our best result slightly exceeds the higher baseline, but the experiments allowed us to draw important considerations regarding the importance of correctly exploiting the visual information and the relevance of the elements present in the memes images.

show abstract

An Interpretable Approach to Hateful Meme Detection

Cited by 10 publications

References 19 publications

Multimodal Hate Speech Detection in Memes Using Contrastive Language-Image Pre-Training

Multimodal Hate Speech Detection in Memes Using Contrastive Language-Image Pre-Training

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

JRLV at SemEval-2022 Task 5: The Importance of Visual Elements for Misogyny Identification in Memes

Contact Info

Product

Resources

About