XED: A Multilingual Dataset for Sentiment Analysis and Emotion Detection

Abstract

We introduce XED, a multilingual fine-grained emotion dataset. The dataset consists of humanannotated Finnish (25k) and English sentences (30k), as well as projected annotations for 30 additional languages, providing new resources for many low-resource languages. We use Plutchik's core emotions to annotate the dataset with the addition of neutral to create a multilabel multiclass dataset. The dataset is carefully evaluated using language-specific BERT models and SVMs to show that XED performs on par with other similar datasets and is therefore a useful tool for sentiment analysis and emotion detection.

Exploring Reliability of Gold Labels for Emotion Detection in Twitter

“…The fairly low agreement among the annotators indicates that the task of detecting Ekman's emotions in tweets is challenging and/or subjective. This is in line with previous studies which reported that emotion detection from text is a complex task that results in low inter-annotator agreements regardless of the emotion taxonomy used (Alm et al, 2005;Schuff et al, 2017;Kim and Klinger, 2018;Bostan and Klinger, 2018;Öhman et al, 2020;Acheampong et al, 2020).…”

“…Several recent surveys (Acheampong et al, 2020;Alswaidan and Menai, 2020) and studies ( Öhman et al, 2020;Bostan et al, 2020;Bostan and Klinger, 2018;Schuff et al, 2017) list previous work on emotion detection from texts and emphasise their differences in type of emotion taxonomy, task (single-label or multi-label), size of the dataset, text genre, granularity, topics, system architectures, and best results obtained with systems for automatic detection of emotions in texts. However, none of the studies focussed on assessing the quality of benchmark datasets, or the influence of methods used for obtaining gold labels on the results of systems for automatic emotion detection from texts.…”

“…Previous studies reported some of the challenges in automatic emotion detection from texts: different perspectives one may take (Buechel and Hahn, 2017b;Alm et al, 2005), missing context ( Öhman et al, 2020;Mohammad, 2012), non-literal meaning (Mohammad, 2012), high subjectivity of the task and low inter-annotator agreement even among trained annotators (Alm et al, 2005;Schuff et al, 2017). For example, the utterance "Italy defeats France in World Cup Final" (Katz et al, 2007) is most probably neutral from the writer's (journalist's) perspective, while evoking strong and probably opposite emotions among Italian and French readers (Buechel and Hahn, 2017b).…”

Continuing Pre-trained Model with Multiple Training Strategies for Emotional Classification

“…As a result, we adopt the continual pre-training method for the masked language model (MLM) (Devlin et al, 2018b) in this Track 2 to directly improve final downstream performance. The available datasets are chosen from the open-source resources (Demszky et al, 2020;Öhman et al, 2020). The optimization function is written as follows…”

The Current State of Finnish NLP

“…There is a Python library called FinMeter 27 (Hämäläinen and Alnajjar, 2019b) that has some higher level semantic tools for Finnish such as metaphor interpretation, word concreteness analysis and sentiment analysis. Sentiment analysis for Finnish has also been studied later on 28 ( Öhman et al, 2020;Vankka et al, 2019;Lindén et al, 2020). There is also research on topic modeling methods (Ginter et al, 2009;Hengchen et al, 2018;Loukasmäki and Makkonen, 2019).…”