MultiModal Sarcasm Detection: A Survey

Bhat, Aruna; Chauhan, Aditya

doi:10.1109/delcon54057.2022.9753058

Cited by 2 publications

(2 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For multimodal sarcasm detection, a recent survey discusses the datasets and approaches in detail (Bhat and Chauhan, 2022). The MUStARD dataset (Castro et al, 2019b) provides clips compiled from popular TV shows, including Friends, The Golden Girls, The Big Bang Theory, and Sarcasmaholics Anonymous, annotated with sarcasm labels.…”

Section: Multimodal Nlpmentioning

confidence: 99%

“…However, sarcasm is also expressed through tonal changes and/or facial expressions. Hence researchers have started investigating modalities other than text, viz., audio and video, to help detect sarcasm (Castro et al, 2019a;Cai et al, 2019;Gupta et al, 2021;Chauhan et al, 2022;Ray et al, 2022). Mishra et al (2017a) observed that gaze features are helpful in detecting sarcasm within short sentences without context, which is our inspiration.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Predict and Use: Harnessing Predicted Gaze to Improve Multimodal Sarcasm Detection

Tiwari,

Kanojia,

Ray

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Sarcasm is a complex linguistic construct with incongruity at its very core. Detecting sarcasm depends on the actual content spoken and tonality, facial expressions, the context of an utterance, and personal traits like language proficiency and cognitive capabilities. In this paper, we propose the utilization of synthetic gaze data to improve the task performance for multimodal sarcasm detection in a conversational setting. We enrich an existing multimodal conversational dataset, i.e., MUStARD++ with gaze features. With the help of human participants, we collect gaze features for < 20% of data instances, and we investigate various methods for gaze feature prediction for the rest of the dataset. We perform extrinsic and intrinsic evaluations to assess the quality of the predicted gaze features. We observe a performance gain of up to 6.6% points by adding a new modality, i.e., collected gaze features. When both collected and predicted data are used, we observe a performance gain of 2.3% points on the complete dataset. Interestingly, with only predicted gaze features, too, we observe a gain in performance (1.9% points). We retain and use the feature prediction model, which maximally correlates with collected gaze features. Our model trained on combining collected and synthetic gaze data achieves SoTA performance on the MUStARD++ dataset. To the best of our knowledge, ours is the first predict-and-use model for sarcasm detection. We publicly release 1 the code, gaze data, and our best models for further research.

show abstract

Section: Multimodal Nlpmentioning

confidence: 99%