Nice Perfume. How Long Did You Marinate in It? Multimodal Sarcasm Explanation

Desai, Poorav; Chakraborty, Tanmoy; Akhtar, Md. Shad

doi:10.1609/aaai.v36i10.21300

Cited by 7 publications

(8 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, we instructed annotators to revise the textual content of the original sample with minimal changes to reverse its label. In addition, to enhance the revising process of the original sarcastic samples, we provided annotators with the explanation of why the sample is sarcastic, which is supplied by the prior work (Desai, Chakraborty, and Akhtar 2022), to facilitate a deeper sarcasm semantics comprehension. Regarding the original non-sarcastic samples, we found that the samples with stronger emotional tendencies are more easily transformed into sarcastic variants.…”

Section: Experiments Experimental Settingsmentioning

confidence: 99%

Debiasing Multimodal Sarcasm Detection with Contrastive Learning

Jia,

Xie,

Jing

2024

AAAI

View full text Add to dashboard Cite

Despite commendable achievements made by existing work, prevailing multimodal sarcasm detection studies rely more on textual content over visual information. It unavoidably induces spurious correlations between textual words and labels, thereby significantly hindering the models' generalization capability. To address this problem, we define the task of out-of-distribution (OOD) multimodal sarcasm detection, which aims to evaluate models' generalizability when the word distribution is different in training and testing settings. Moreover, we propose a novel debiasing multimodal sarcasm detection framework with contrastive learning, which aims to mitigate the harmful effect of biased textual factors for robust OOD generalization. In particular, we first design counterfactual data augmentation to construct the positive samples with dissimilar word biases and negative samples with similar word biases. Subsequently, we devise an adapted debiasing contrastive learning mechanism to empower the model to learn robust task-relevant features and alleviate the adverse effect of biased words. Extensive experiments show the superiority of the proposed framework.

show abstract

Section: Experiments Experimental Settingsmentioning

confidence: 99%

Debiasing Multimodal Sarcasm Detection with Contrastive Learning

Jia,

Xie,

Jing

2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Delving further, Kumar et al (2022) delve into dissecting the discourse structure within sarcastic conversations and introduce the Sarcasm Explanation in Dialogue (SED) task. In a similar vein, Desai, Chakraborty, and Akhtar (2022) bring to the fore an innovative challenge named Multimodal Sarcasm Explanation (MuSE), revolving around generating a natural language explanation for a multimodal sarcastic expression encompassing both an image and a caption.…”

Section: Related Workmentioning

confidence: 99%

“…In (Desai, Chakraborty, and Akhtar 2022), the authors introduce a new task known as Multimodal Sarcasm Explanation (MuSE). In this problem, the objective is to provide a natural language explanation for a multimodal sarcastic post, which comprises both an image and a caption.…”

Section: Related Workmentioning

confidence: 99%

Well, Now We Know! Unveiling Sarcasm: Initiating and Exploring Multimodal Conversations with Reasoning

Singh,

Firdaus,

Chauhan

et al. 2024

AAAI

View full text Add to dashboard Cite

Sarcasm is a widespread linguistic phenomenon that poses a considerable challenge to explain due to its subjective nature, absence of contextual cues, and rooted personal perspectives. Even though the identification of sarcasm has been extensively studied in dialogue analysis, merely detecting sarcasm falls short of enabling conversational systems to genuinely comprehend the underlying meaning of a conversation and generate fitting responses. It is imperative to not only detect sarcasm but also pinpoint its origination and the rationale behind the sarcastic expressions to capture its authentic essence. In this paper, we delve into the discourse structure of conversations infused with sarcasm and introduce a novel task - Sarcasm Initiation and Reasoning in Conversations (SIRC). Embedded in a multimodal environment and involving a combination of both English and code-mixed interactions, the objective of the task is to discern the trigger or starting point of sarcasm. Additionally, the task involves producing a natural language explanation that rationalizes the satirical dialogues. To this end, we introduce Sarcasm Initiation and Reasoning Dataset (SIRD) to facilitate our task and provide sarcasm initiation annotations and reasoning. We develop a comprehensive model named Sarcasm Initiation and Reasoning Generation (SIRG), which is designed to encompass textual, audio, and visual representations. To achieve this, we introduce a unique shared fusion method that employs cross-attention mechanisms to seamlessly integrate these diverse modalities. Our experimental outcomes, conducted on the SIRC dataset, demonstrate that our proposed framework establishes a new benchmark for both sarcasm initiation and its reasoning generation in the context of multimodal conversations. The code and dataset can be accessed from https://www.iitp.ac.in/∼ai-nlp-ml resources.html#sarcasm-explain and https://github.com/GussailRaat/SIRG-Sarcasm-Initiation-and-Reasoning-Generation.

show abstract

“…However, figurative meaning is also prevalent in visual phenomena, such as visual metaphors (Akula et al, 2023;Chakrabarty et al, 2023), multimodal sarcasm (Desai et al, 2022), and humor (Hessel et al, 2023;Hwang and Shwartz, 2023). Yet so far most of the work on vision and language models (VLMs) has focused on understanding literal meaning in images and captions (e.g., ScienceQA , MMMU (Yue et al, 2024)) including work on explainable visual entailment (Kayser et al, 2021).…”

Section: Image (Premise)mentioning

confidence: 99%

FLUTE: Figurative Language Understanding through Textual Explanations

Chakrabarty¹,

Saakyan²,

Ghosh³

et al. 2022

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Large Vision-Language models (VLMs) have demonstrated strong reasoning capabilities in tasks requiring a fine-grained understanding of literal images and text, such as visual question-answering or visual entailment. However, there has been little exploration of these models' capabilities when presented with images and captions containing figurative phenomena such as metaphors or humor, the meaning of which is often implicit. To close this gap, we propose a new task and a high-quality dataset: Visual Figurative Language Understanding with Textual Explanations (V-FLUTE). We frame the visual figurative language understanding problem as an explainable visual entailment task, where the model has to predict whether the image (premise) entails a claim (hypothesis) and justify the predicted label with a textual explanation. Using a human-AI collaboration framework, we build a highquality dataset, V-FLUTE, that contains 6,027 instances spanning five diverse multimodal figurative phenomena: metaphors, similes, idioms, sarcasm, and humor. The figurative phenomena can be present either in the image, the caption, or both. We further conduct both automatic and human evaluations to assess current VLMs' capabilities in understanding figurative phenomena. 1

show abstract

Nice Perfume. How Long Did You Marinate in It? Multimodal Sarcasm Explanation

Cited by 7 publications

References 26 publications

Debiasing Multimodal Sarcasm Detection with Contrastive Learning

Debiasing Multimodal Sarcasm Detection with Contrastive Learning

Well, Now We Know! Unveiling Sarcasm: Initiating and Exploring Multimodal Conversations with Reasoning

FLUTE: Figurative Language Understanding through Textual Explanations

Contact Info

Product

Resources

About