We propose our solution to the multimodal semantic role labeling task from the CON-STRAINT'22 workshop. The task aims at classifying entities in memes into classes such as "hero" and "villain". We use several pre-trained multi-modal models to jointly encode the text and image of the memes, and implement three systems to classify the role of the entities. We propose dynamic sampling strategies to tackle the issue of class imbalance. Finally, we perform qualitative analysis on the representations of the entities. * These authors contributed equally. 1 We take each (meme, entity) pair as independent sam-

No abstract