MMVAE at SemEval-2022 Task 5: A Multi-modal Multi-task VAE on Misogynous Meme Detection

Abstract

Memes have become quite common in day-today communications on social media platforms. They often appear to be amusing, evoking and attractive to audiences. However, some memes containing malicious content can be harmful to targeted groups. In this paper, we study misogynous meme detection, a shared task in SemEval 2022 -Multimedia Automatic Misogyny Identification (MAMI). The challenge of misogynous meme detection is to co-represent multi-modal features. To tackle with this challenge, we propose a Multi-modal Multi-task Variational Au-toEncoder (MMVAE) to learn an effective corepresentation of visual and textual features in the latent space. Our goal is to automatically determine if a meme contains misogynous information and then identify its fine-grained category. Our model achieves F 1 scores of 0.723 on the MAMI sub-task A and 0.634 on sub-task B. We carry out comprehensive experiments on our model's architecture and show that our approach significantly outperforms several strong uni-modal and multi-modal approaches. Our code is released on github 1 .

DeVi Deep Learning Framework for Misogyny Identification in Multimodal Data

“…For text, GloVe embeddings are used to initialize individual words and pass this sequence through a layer of deep learning model, LSTM. A multi-modal multi-task variational autoencoder (MMVAE) is discussed by Gu et al [37] designed to integrate multimodal features. The image embedding of the meme was obtained through a sequence of trials utilizing two distinct pre-trained models: ResNet-50 and OpenAI CLIP-ViT-B32.…”

SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification

“…Leaderboard Sub-task A (Srivastava, 2022) 0.759 R2D2* (Sharma et al, 2022b) 0.757 PAIC (ZHI et al, 2022) 0.755 ymf924 0.755 RubCSG* (Yu et al, 2022) 0.755 hate-alert 0.753 AMS_ADRN* (Li et al, 2022) 0.746 TIB-VA* (Hakimov et al, 2022) 0.734 4 union 0.727 Unibo* (Muti et al, 2022) 0.727 MMVAE* (Gu et al, 2022b) 0.723 YMAI* (Habash et al, 2022) 0.722 Transformers* (Mahadevan et al, 2022) 0.718 taochen* (Tao and jae Kim, 2022) 0.716 codec* (Mahran et al, 2022) 0.715 QMUL* 0.714 UPB* (Paraschiv et al, 2022) 0.714 HateU* (Arango et al, 2022) 0.712 yuanyuanya 0.708 Triplo7* (Attanasio et al, 2022) 0.699 InfUfrgs* (Lorentz and Moreira, 2022) 0.698 Mitra Behzadi* (Behzadi et al, 2022) 0.694 Gini_us* 0.692 5 riziko 0.687 UMUTeam* (García-Díaz et al, 2022) 0.687 Tathagata Raha* (Raha et al, 2022) 0.687 LastResort* (Agrawal and Mamidi, 2022) 0.686 TeamOtter* (Maheshwari and Nangi, 2022) 0.679 ShailyDesai 0.677 JRLV* (Ravagli and Vaiani, 2022) 0.670 I2C* (Cordon et al, 2022) 0.665 qinian* (Gu et al, 2022a) 0…”