BLUE at Memotion 2.0 2022: You have my Image, my Text and my Transformer

Abstract

Memes are prevalent on the internet and continue to grow and evolve alongside our culture. An automatic understanding of memes propagating on the internet can shed light on the general sentiment and cultural attitudes of people. In this work, we present team BLUE's solution for the second edition of the MEMOTION shared task. We showcase two approaches for meme classification (i.e. sentiment, humour, offensive, sarcasm and motivation levels) using a text-only method using BERT, and a Multi-Modal-Multi-Task transformer network that operates on both the meme image and its caption to output the final scores. In both approaches, we leverage state-of-the-art pretrained models for text (BERT, Sentence Transformer) and image processing (EfficientNetV4, CLIP). Through our efforts, we obtain first place in task A, second place in task B and third place in task C. In addition, our team obtained the highest average score for all three tasks.

The contingent macro

“…Giorgi (2021), following an assessment of the literature on memes, proposes to identify memes based on three features: irony, manipulation, and intertextuality. Research on memes also witnessed a rising trend in studies employing computational techniques for the automated recognition and classification of memetic instances (Miliani et al, 2020;Bucur et al, 2022;Theisen et al, 2021). Here, too, memes are identified and sorted depending on the presence of characteristics, like multimodality or recurrent layouts.…”

Fine-tuning and Sampling Strategies for Multimodal Role Labeling of Entities under Class Imbalance

“…We extract features using FasterRCNN (Ren et al, 2015), EfficientNet (Tan and Le, 2019) and VGG (Simonyan and Zisserman, 2015). Bucur et al (2022) showed that EfficientNet features prove useful for sentiment and emotion analyses of meme, while Pramanick et al (2021) prove the efficiency of VGG for detecting harmful memes and identifying their target.…”