JRLV at SemEval-2022 Task 5: The Importance of Visual Elements for Misogyny Identification in Memes

Abstract

Gender discrimination is a serious and widespread problem on social media and online in general. Besides offensive messages, memes are one of the main means of dissemination for such content. With these premises, the MAMI task was proposed at the SemEval-2022, which consists of identifying memes with misogynous characteristics. In this work, we propose a solution to this problem based on Mask R-CNN and VisualBERT that leverages the multimodal nature of the task. Our study focuses on observing how the two sources of data in memes (text and image) and their possible combinations impact performances. Our best result slightly exceeds the higher baseline, but the experiments allowed us to draw important considerations regarding the importance of correctly exploiting the visual information and the relevance of the elements present in the memes images.

DeVi Deep Learning Framework for Misogyny Identification in Multimodal Data

“…A total of six unique ensemble models were created, with the BiLSTM Neural Network, incorporating BERT embeddings and employing a threshold of 0.45 for nude detection, emerging as the definitive model. Ravagli et al [44] discussed a technique that integrates Mask R-CNN and VisualBERT models to effectively address the multimodal requirements of the task outlined in the MAMI challenge.…”

SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification

“…Leaderboard Sub-task A (Srivastava, 2022) 0.759 R2D2* (Sharma et al, 2022b) 0.757 PAIC (ZHI et al, 2022) 0.755 ymf924 0.755 RubCSG* (Yu et al, 2022) 0.755 hate-alert 0.753 AMS_ADRN* (Li et al, 2022) 0.746 TIB-VA* (Hakimov et al, 2022) 0.734 4 union 0.727 Unibo* (Muti et al, 2022) 0.727 MMVAE* (Gu et al, 2022b) 0.723 YMAI* (Habash et al, 2022) 0.722 Transformers* (Mahadevan et al, 2022) 0.718 taochen* (Tao and jae Kim, 2022) 0.716 codec* (Mahran et al, 2022) 0.715 QMUL* 0.714 UPB* (Paraschiv et al, 2022) 0.714 HateU* (Arango et al, 2022) 0.712 yuanyuanya 0.708 Triplo7* (Attanasio et al, 2022) 0.699 InfUfrgs* (Lorentz and Moreira, 2022) 0.698 Mitra Behzadi* (Behzadi et al, 2022) 0.694 Gini_us* 0.692 5 riziko 0.687 UMUTeam* (García-Díaz et al, 2022) 0.687 Tathagata Raha* (Raha et al, 2022) 0.687 LastResort* (Agrawal and Mamidi, 2022) 0.686 TeamOtter* (Maheshwari and Nangi, 2022) 0.679 ShailyDesai 0.677 JRLV* (Ravagli and Vaiani, 2022) 0.670 I2C* (Cordon et al, 2022) 0.665 qinian* (Gu et al, 2022a) 0…”