Beyond Emotion: A Multi-Modal Dataset for Human Desire Understanding

Jia, Ao; He, Ya‐Ling; Zhang, Yazhou; Uprety, Sagar; Song, Dandan; Lioma, Christina

doi:10.18653/v1/2022.naacl-main.108

Cited by 8 publications

(2 citation statements)

References 23 publications

(24 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The representative datasets contain MELD, IEMOCAP, etc. For example, Jia et al (2022) constructed a multi-modal emotion and desire recognition dataset, called MSED. It consisted of 9,190 text-image pairs collected from a wide range of social media resources, e.g., Twitter, Getty Image, Flickr.…”

Section: Multi-modal Emotion Recognitionmentioning

confidence: 99%

Multi-Modal Attentive Prompt Learning for Few-shot Emotion Recognition in Conversations

Liang,

Tu,

et al. 2024

jair

View full text Add to dashboard Cite

Emotion recognition in conversations (ERC) has emerged as an important research area in Natural Language Processing and Affective Computing, focusing on accurately identifying emotions within the conversational utterance. Conventional approaches typically rely on labeled training samples for fine-tuning pre-trained language models (PLMs) to enhance classification performance. However, the limited availability of labeled data in real-world scenarios poses a significant challenge, potentially resulting in diminished model performance. In response to this challenge, we present the Multi-modal Attentive Prompt (MAP) learning framework, tailored specifically for few-shot emotion recognition in conversations. The MAP framework consists of four integral modules: multi-modal feature extraction for the sequential embedding of text, visual, and acoustic inputs; a multi-modal prompt generation module that creates six manually-designed multi-modal prompts; an attention mechanism for prompt aggregation; and an emotion inference module for emotion prediction. To evaluate our proposed model’s efficacy, we conducted extensive experiments on two widely recognized benchmark datasets, MELD and IEMOCAP. Our results demonstrate that the MAP framework outperforms state-of-the-art ERC models, yielding notable improvements of 3.5% and 0.4% in micro F1 scores. These findings highlight the MAP learning framework’s ability to effectively address the challenge of limited labeled data in emotion recognition, offering a promising strategy for improving ERC model performance.

show abstract

Section: Multi-modal Emotion Recognitionmentioning

confidence: 99%

Multi-Modal Attentive Prompt Learning for Few-shot Emotion Recognition in Conversations

Liang,

Tu,

et al. 2024

jair

View full text Add to dashboard Cite

show abstract

“…Intentions have been extensively explored in psychology tests, e.g., behavioral re-enactment (Meltzoff, 1995), action prediction (Phillips et al, 2002, intention explanation (Smiley, 2001), and intention attribution to abstract figures (Castelli, 2006). EPISTEMIC REASONING (Cohen, 2021) Infer T -✓ ✓ TOMI (Nematzadeh et al, 2018) QA T -✓ ✓ ✓ HI-TOM (He et al, 2023) QA T -✓ ✓ ✓ MINDGAMES (Sileo and Lernould, 2023) Infer T -✓ ✓ ✓ ✓ ADV-CSFB (Shapira et al, 2023a) QA H -✓ ✓ CONVENTAIL (Zhang and Chai, 2010) Infer H -✓ ✓ ✓ ✓ SOCIALIQA (Sap et al, 2019) QA H -✓ ✓ ✓ BEST (Tracey et al, 2022) -H -✓ ✓ ✓ ✓ FAUXPAS-EAI (Shapira et al, 2023b) QA H,AI -✓ ✓ ✓ COKE NLG AI -✓ ✓ ✓ ✓ TOM-IN-AMC Infer H -✓ ✓ ✓ ✓ G4C (Zhou et al, 2023b) NLG H,AI -✓ ✓ ✓ ✓ ✓ ✓ VISUALBELIEFS (Eysenbach et al, 2016) Infer -Cartoon ✓ ✓ ✓ TRIANGLE COPA (Gordon, 2016) QA H Cartoon ✓ ✓ ✓ ✓ MSED (Jia et al, 2022) Infer H Images ✓ ✓ ✓ BIB (Gandhi et al, 2021) Infer -2D Grid ✓ ✓ ✓ AGENT (Shu et al, 2021) Infer -3D Sim. ✓ ✓ ✓ ✓ MTOM (Rabinowitz et al, 2018) Infer -2D Grid ✓ ✓ ✓ SYMMTOM (Sclar et al, 2022) MARL -2D Grid ✓ ✓ ✓ ✓ ✓ ✓ MINDCRAFT (Bara et al, 2021) Infer (Bara et al, 2023) Infer Desires.…”

Section: Abilities In Theory Of Mind Space (Atoms) Frameworkmentioning

confidence: 99%

Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models

Ma,

Sansom,

Peng

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Large Language Models (LLMs) have generated considerable interest and debate regarding their potential emergence of Theory of Mind (ToM). Several recent inquiries reveal a lack of robust ToM in these models and pose a pressing demand to develop new benchmarks, as current ones primarily focus on different aspects of ToM and are prone to shortcuts and data leakage. In this position paper, we seek to answer two road-blocking questions: (1) How can we taxonomize a holistic landscape of machine ToM? (2) What is a more effective evaluation protocol for machine ToM? Following psychological studies, we taxonomize machine ToM into 7 mental state categories and delineate existing benchmarks to identify under-explored aspects of ToM. We argue for a holistic and situated evaluation of ToM to break ToM into individual components and treat LLMs as an agent who is physically situated in environments and socially situated in interactions with humans. Such situated evaluation provides a more comprehensive assessment of mental states and potentially mitigates the risk of shortcuts and data leakage. We further present a pilot study in a grid world setup as a proof of concept. We hope this position paper can facilitate future research to integrate ToM with LLMs and offer an intuitive means for researchers to better position their work in the landscape of ToM. of faux pas by normally developing children and children with asperger syndrome or high-functioning autism. Journal of autism and developmental disorders, 29:407-418.Cindy Beaudoin, Élizabel Leblanc, Charlotte Gagner, and Miriam H Beauchamp. 2020. Systematic review and inventory of theory of mind measures for young children. Frontiers in psychology, 10:2905.

show abstract