Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features

Karthik, Kumar, Gokul; Nandakumar, Karthik

doi:10.18653/v1/2022.nlp4pi-1.20

Cited by 9 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The solution's broad knowledge base, while extensive, is also less efficient compared to specialized models like QLoRA [2] or HateClipper [4], which fine-tune knowledge more effectively for specific tasks. In contrast however, this paper suggests distilling large VLMs into simpler forms as a viable strategy when detailed, task-specific data is unavailable.…”

Section: Discussion and Limitationsmentioning

confidence: 99%

“…Llava's innovation to increase the input dimensions of images and allocate more tokens for encoding visual information significantly enhanced model performance. Using OpenAI's 4 CLIP technology, they managed to encode images up to 672x672 pixels from an original 336x336, marking a substantial improvement. Local testing on the MultiOFF dataset [10] with the adjusted resolution, resulted in a 6% improvement in AUROC, showcasing a direct correlation between enhanced image resolution and better meme understanding.…”

Section: Natural Language Understanding and Visual Understandingmentioning

confidence: 99%

See 1 more Smart Citation

OSPC: Artificial VLM Features for Hateful Meme Detection

Grönquist

2024

Companion Proceedings of the ACM Web Conference 2024

View full text Add to dashboard Cite

The digital revolution and the advent of the world wide web have transformed human communication, notably through the emergence of memes. While memes are a popular and straightforward form of expression, they can also be used to spread misinformation and hate due to their anonymity and ease of use. In response to these challenges, this paper introduces a solution developed by team 'Baseline' for the AI Singapore Online Safety Prize Challenge. Focusing on computational efficiency and feature engineering, the solution achieved an AUROC of 0.76 and an accuracy of 0.69 on the test dataset. As key features, the solution leverages the inherent probabilistic capabilities of large Vision-Language Models (VLMs) to generate task-adapted feature encodings from text, and applies a distilled quantization tailored to the specific cultural nuances present in Singapore. This type of processing and fine-tuning can be adapted to various visual and textual understanding and classification tasks, and even applied on private VLMs such as OpenAI's GPT. Finally it can eliminate the need for extensive model training on large GPUs for resource constrained applications, also offering a solution when little or no data is available.

show abstract

Section: Discussion and Limitationsmentioning

confidence: 99%

Section: Natural Language Understanding and Visual Understandingmentioning

confidence: 99%