Creating a Dataset for Multilingual Fine-grained Emotion-detection Using Gamification-based Annotation

Öhman, Emily; Kajava, Kaisla; Tiedemann, Jörg; Honkela, Timo

doi:10.18653/v1/w18-6205

Cited by 13 publications

(3 citation statements)

References 14 publications

(12 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3 Human Evaluation. We perform a human evaluation with our proposed metric, which is absent in previous work for measuring inter-annotator agreement for emotion annotations (Wood et al, 2018;Öhman et al, 2018). Crowdsourced workers are asked to determine the agreement between two annotation pairs constructed from three annotators, that is, A: (e 1 , e 2 ) and B: (e 1 , e 3 ).…”

Section: Inter-annotator Agreementmentioning

confidence: 99%

Detecting Perceived Emotions in Hurricane Disasters

Desai¹,

Caragea²,

Li³

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Natural disasters (e.g., hurricanes) affect millions of people each year, causing widespread destruction in their wake. People have recently taken to social media websites (e.g., Twitter) to share their sentiments and feelings with the larger community. Consequently, these platforms have become instrumental in understanding and perceiving emotions at scale. In this paper, we introduce HURRICANEEMO, an emotion dataset of 15,000 English tweets spanning three hurricanes: Harvey, Irma, and Maria. We present a comprehensive study of fine-grained emotions and propose classification tasks to discriminate between coarsegrained emotion groups. Our best BERT (Devlin et al., 2019) model, even after task-guided pre-training which leverages unlabeled Twitter data, achieves only 68% accuracy (averaged across all groups). HURRICANEEMO serves not only as a challenging benchmark for models but also as a valuable resource for analyzing emotions in disaster-centric domains. Related WorkEmotion detection has been extensively studied in news headlines (Strapparava and Mihalcea, 2007;

show abstract

Section: Inter-annotator Agreementmentioning

confidence: 99%

Detecting Perceived Emotions in Hurricane Disasters

Desai¹,

Caragea²,

Li³

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Previous work on multilingual style predominantly focuses on training LMs to perform cross-lingual and multilingual style classification and style transfer. Key styles studied include formality (Briakou et al, 2021;Krishna et al, 2022;Rippeth et al, 2022) and emotion (Öhman et al, 2018;Lamprinidis et al, 2021;Öhman et al, 2020), with another body of work focusing on style-aware multilingual generation with any subset of chosen styles Garcia et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

Comparing Styles across Languages

Havaldar,

Pressimone,

Wong

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Understanding how styles differ across languages is advantageous for training both humans and computers to generate culturally appropriate text. We introduce an explanation framework to extract stylistic differences from multilingual LMs and compare styles across languages. Our framework (1) generates comprehensive style lexica in any language and (2) consolidates feature importances from LMs into comparable lexical categories. We apply this framework to compare politeness, creating the first holistic multilingual politeness dataset and exploring how politeness varies across four languages. Our approach enables an effective evaluation of how distinct linguistic categories contribute to stylistic variations and provides interpretable insights into how people communicate differently around the world.

show abstract

“…Gamification techniques can be added to data labeling, evaluation, and production. Indeed, some NLP work is using gamification, mostly for data collection (e.g., Kumaran et al, 2014;Ogawa et al, 2020;Öhman et al, 2018).…”

Section: Triggering Addictive Behaviourmentioning

confidence: 99%

Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing

Shmueli¹,

Fell²,

Ray³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

The use of crowdworkers in NLP research is growing rapidly, in tandem with the exponential increase in research production in machine learning and AI. Ethical discussion regarding the use of crowdworkers within the NLP research community is typically confined in scope to issues related to labor conditions such as fair pay. We draw attention to the lack of ethical considerations related to the various tasks performed by workers, including labeling, evaluation, and production. We find that the Final Rule, the common ethical framework used by researchers, did not anticipate the use of online crowdsourcing platforms for data collection, resulting in gaps between the spirit and practice of human-subjects ethics in NLP research. We enumerate common scenarios where crowdworkers performing NLP tasks are at risk of harm. We thus recommend that researchers evaluate these risks by considering the three ethical principles set up by the Belmont Report. We also clarify some common misconceptions regarding the Institutional Review Board (IRB) application. We hope this paper will serve to reopen the discussion within our community regarding the ethical use of crowdworkers.

show abstract

Creating a Dataset for Multilingual Fine-grained Emotion-detection Using Gamification-based Annotation

Cited by 13 publications

References 14 publications

Detecting Perceived Emotions in Hurricane Disasters

Detecting Perceived Emotions in Hurricane Disasters

Comparing Styles across Languages

Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing

Contact Info

Product

Resources

About