IIITH at SemEval-2022 Task 5: A comparative study of deep learning models for identifying misogynous memes

Abstract

This paper provides a comparison of different deep learning methods for identifying misogynous memes for SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification. In this task, we experiment with architectures in the identification of misogynous content in memes by making use of text and image-based information. The different deep learning methods compared in this paper are: (i) unimodal image or text models (ii) fusion of unimodal models (iii) multimodal transformers models and (iv) transformers further pretrained on a multimodal task. From our experiments, we found pretrained multimodal transformer architectures to strongly outperform the models involving fusion of representation from both the modalities.

SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification

“…Table 10 Continued from previous page Leaderboard Sub-task B Team Name Weighted-average F 1 -score 13 TeamOtter* (Maheshwari and Nangi, 2022) 0.680 14 Tathagata Raha* (Raha et al, 2022) 0.679 15 UPB* (Paraschiv et al, 2022) 0…”

“…Leaderboard Sub-task A (Srivastava, 2022) 0.759 R2D2* (Sharma et al, 2022b) 0.757 PAIC (ZHI et al, 2022) 0.755 ymf924 0.755 RubCSG* (Yu et al, 2022) 0.755 hate-alert 0.753 AMS_ADRN* (Li et al, 2022) 0.746 TIB-VA* (Hakimov et al, 2022) 0.734 4 union 0.727 Unibo* (Muti et al, 2022) 0.727 MMVAE* (Gu et al, 2022b) 0.723 YMAI* (Habash et al, 2022) 0.722 Transformers* (Mahadevan et al, 2022) 0.718 taochen* (Tao and jae Kim, 2022) 0.716 codec* (Mahran et al, 2022) 0.715 QMUL* 0.714 UPB* (Paraschiv et al, 2022) 0.714 HateU* (Arango et al, 2022) 0.712 yuanyuanya 0.708 Triplo7* (Attanasio et al, 2022) 0.699 InfUfrgs* (Lorentz and Moreira, 2022) 0.698 Mitra Behzadi* (Behzadi et al, 2022) 0.694 Gini_us* 0.692 5 riziko 0.687 UMUTeam* (García-Díaz et al, 2022) 0.687 Tathagata Raha* (Raha et al, 2022) 0.687 LastResort* (Agrawal and Mamidi, 2022) 0.686 TeamOtter* (Maheshwari and Nangi, 2022) 0.679 ShailyDesai 0.677 JRLV* (Ravagli and Vaiani, 2022) 0.670 I2C* (Cordon et al, 2022) 0.665 qinian* (Gu et al, 2022a) 0…”