WASSA 2022 Shared Task: Predicting Empathy, Emotion and Personality in Reaction to News Stories

Barrière, Valentin; Tafreshi, Shabnam; Sedoc, João; Alqahtani, Sawsan

doi:10.18653/v1/2022.wassa-1.20

Cited by 6 publications

(9 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Pearson correlation between the human evaluations and our predicted score is 0.72. We have seen that the Pearson correlation scores produced (Barriere et al, 2022). by our approach are comparatively higher as compared to the previous studies done on humanannotated scores for learning.…”

Section: Resultscontrasting

confidence: 56%

“…by our approach are comparatively higher as compared to the previous studies done on humanannotated scores for learning. SINAI, CAISA, and LingJing are teams belonging to WASSA 2022 shared task (Barriere et al, 2022) (Table 4). We tested six models during inference time.…”

Section: Resultsmentioning

confidence: 99%

“…However, measuring empathy in open-domain dialogues is a challenging task, as it requires understanding the context and emotions of the conversation. In recent years, there has been a growing interest in developing automatic methods for measuring empathy in open-domain dialogues (Barriere et al, 2022) (Hosseini and Caragea, 2021). Traditional methods for measuring empathy rely on human evaluations, which can be time-consuming and costly.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

EMP-EVAL: A Framework for Measuring Empathy in Open Domain Dialogues

Amjad¹,

Zeeshan²,

Beg³

2023

Preprint

View full text Add to dashboard Cite

Measuring empathy in conversation can be challenging, as empathy is a complex and multifaceted psychological construct that involves both cognitive and emotional components. Human evaluations can be subjective, leading to inconsistent results. Therefore, there is a need for an automatic method for measuring empathy that reduces the need for human evaluations. In this paper, we proposed a novel approach EMP-EVAL, a simple yet effective automatic empathy evaluation method. The proposed technique takes the influence of Emotion, Cognitive and Emotional empathy. To the best knowledge, our work is the first to systematically measure empathy without the humanannotated provided scores. Experimental results demonstrate that our metrics can correlate with human preference, achieving comparable results with human judgments.

show abstract

Section: Resultscontrasting

confidence: 56%

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

EMP-EVAL: A Framework for Measuring Empathy in Open Domain Dialogues

Amjad¹,

Zeeshan²,

Beg³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…A common approach is to predict the emotion or persona from a pre-defined set and generate the response in a multi-task manner (Ma et al, 2021;Sabour et al, 2022;. Besides that, lots of work notices these linguistic cues underneath text by directly predicting them independently as a classification task Barriere et al, 2022;Ghosh et al, 2022). Distinguishing from these previous works, we regard different aspects of cues as part of user status and prompt the LLMs to reason user status exhibited in the dialogue context, aiming to generate more helpful and acceptable responses for users.…”

Section: Related Workmentioning

confidence: 99%

“…Personality. Previous works found that the content and style of a user's inquiry can provide indirect insights into their personality traits (Mairesse et al, 2007;Barriere et al, 2022). For instance, an individual with a tendency towards anxiety may ask for advice on how to alleviate nervousness before an upcoming job interview, phrasing the question as follows: "What strategies can I employ to reduce my anxiety and perform well in tomorrow's interview?".…”

Section: Datasets Collectionmentioning

confidence: 99%

Cue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMs

Wang,

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Large Language Models (LLMs), such as ChatGPT, greatly empower dialogue systems with strong language understanding and generation capabilities. However, most of the previous works prompt the LLMs to directly generate a response based on the dialogue context, overlooking the underlying linguistic cues about the user status exhibited in the context. Such in-depth dialogue scenarios are challenging for existing LLMs to figure out the user's hidden needs and respond satisfactorily through a single-step inference. To this end, we propose a novel linguistic cue-based chain-of-thoughts (Cue-CoT), which enhances the LLMs inference with an intermediate reasoning step to find cues exhibited in the dialogue, aiming to provide a more personalized and engaging response. To evaluate the approach, we build a benchmark with in-depth dialogue questions, consisting of 6 datasets in both Chinese and English, targeting 3 major linguistic cues during the conversation: personality, emotion, and psychology. We conduct extensive experiments on the proposed benchmark with 5 LLMs under both zero-shot and one-shot settings. Empirical results demonstrate our proposed Cue-CoT method outperforms standard prompting methods in terms of both helpfulness and acceptability on all datasets.

show abstract