Crowdsourcing a Word–emotion Association Lexicon

Abstract

Even though considerable attention has been given to the polarity of words (positive and negative) and the creation of large polarity lexicons, research in emotion analysis has had to rely on limited and small emotion lexicons. In this paper we show how the combined strength and wisdom of the crowds can be used to generate a large, high-quality, word-emotion and word-polarity association lexicon quickly and inexpensively. We enumerate the challenges in emotion annotation in a crowdsourcing scenario and propose solutions to address them. Most notably, in addition to questions about emotions associated with terms, we show how the inclusion of a word choice question can discourage malicious data entry, help identify instances where the annotator may not be familiar with the target term (allowing us to reject such annotations), and help obtain annotations at sense level (rather than at word level). We conducted experiments on how to formulate the emotionannotation questions, and show that asking if a term is associated with an emotion leads to markedly higher inter-annotator agreement than that obtained by asking if a term evokes an emotion.

Lexicon based feature extraction for emotion text classification

“…Whilst GPELs offer useful knowledge about emotion-rich words, they are static and are likely to have poor coverage of the emotion vocabulary used in domains like Twitter. For emotion classification of tweets, Mohammad [17] and [22] demonstrated that DSEL based features offer significant gains over n-grams when compared to those of GPEL based features [32]. However feature extraction using DSELs has not been explored beyond binary and integer counts.…”

“…Further, unfair may be associated with anger despite being more dominant in sadness related documents; the crisp binary memberships of words in GPELs do not allow to capture such fuzzy memberships of words to emotion classes, thereby making them limitedly effective for feature extraction. Accordingly, recent efforts in emotion analysis focused on learning domain specific lexicons [20,21] and also utilizing them for emotion feature extraction [17,22]. However the emotion features extracted were limited to simple emotion word counts in a document using the lexicon, which, while being simple, do not exploit the knowledge of the lexicon in its entirety.…”

“…Such word-emotion associations are usually captured by emotion lexicons. Existing general purpose emotion lexicons (GPELs) such as WordNet-Affect (WNA) [15], EmoSenticNet (ESN) [16] and NRC word-emotion lexicon [17], which are hand crafted, associate between words and emotions identified by Ekman and Plutchik. Emotion features extracted using the knowledge of the GPELs, when combined with traditional BoW features improved emotion classification significantly [18,19].…”

Crowdsourcing for web genre annotation

“…Since in Mturk the annotations have been done by various workers, Cohen's j is not applicable as it needs a consistent set of annotators for all items. Therefore, like other annotation studies using crowd-sourcing (Mohammad and Turney 2012;McCreadie et al 2011;Bentivogli et al 2011), we calculated Fleiss's kappa (Fleiss 1971) for the annotation. The next section presents inter-coder agreement results.…”

A model for providing emotion awareness and feedback using fuzzy logic in online learning

“…The NRC Emotion Lexicon (EmoLex) is a list of English words and their associations with eight basic emotions (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and two sentiments (negative and positive) (Mohammad & Turney 2013). The annotations were manually done by crowd sourcing.…”