Case Study: The Manually Annotated Sub-Corpus

Ide, Nancy

doi:10.1007/978-94-024-0881-2_19

Cited by 32 publications

(40 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We chose the Manually Annotated Sub-Corpus (MASC) of the American National Corpus which is already annotated for various linguistic levels (Ide et al, 2008;Ide et al, 2010). We excluded registers containing spoken, mainly dialogic or non-standard language, e.g., telephone conversations, movie scripts and tweets.…”

Section: Methodsmentioning

confidence: 99%

Readers vs. Writers vs. Texts: Coping with Different Perspectives of Text Understanding in Emotion Annotation

Buechel

Hahn

2017

Proceedings of the 11th Linguistic Annotation Workshop

View full text Add to dashboard Cite

We here examine how different perspectives of understanding written discourse, like the reader's, the writer's or the text's point of view, affect the quality of emotion annotations. We conducted a series of annotation experiments on two corpora, a popular movie review corpus and a genreand domain-balanced corpus of standard English. We found statistical evidence that the writer's perspective yields superior annotation quality overall. However, the quality one perspective yields compared to the other(s) seems to depend on the domain the utterance originates from. Our data further suggest that the popular movie review data set suffers from an atypical bimodal distribution which may decrease model performance when used as a training resource.

show abstract

Section: Methodsmentioning

confidence: 99%

Readers vs. Writers vs. Texts: Coping with Different Perspectives of Text Understanding in Emotion Annotation

Buechel

Hahn

2017

Proceedings of the 11th Linguistic Annotation Workshop

View full text Add to dashboard Cite

show abstract

“…Following these criteria, we composed our corpus out of several categories of the Manually Annotated Sub-Corpus of the American National Corpus (MASC; Ide et al (2008), Ide et al (2010)) and the corpus of SemEval-2007 Task 14 Affective Text (SE07; Strapparava and Mihalcea (2007)). MASC is already annotated on various linguistic levels.…”

Section: Corpus Design and Creationmentioning

confidence: 99%

EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis

Buechel

Hahn

2017

Proceedings of the 15th Conference of the European Chapter of The Association for Computational Linguistics: Volume 2

157

103

View full text Add to dashboard Cite

We describe EMOBANK, a corpus of 10k English sentences balancing multiple genres, which we annotated with dimensional emotion metadata in the Valence-ArousalDominance (VAD) representation format. EMOBANK excels with a bi-perspectival and bi-representational design. On the one hand, we distinguish between writer's and reader's emotions, on the other hand, a subset of the corpus complements dimensional VAD annotations with categorical ones based on Basic Emotions. We find evidence for the supremacy of the reader's perspective in terms of IAA and rating intensity, and achieve close-to-human performance when mapping between dimensional and categorical formats.

show abstract

“…For benchmarking the NER systems, we use the Manually Annotated SubCorpus (MASC) (Ide et al, 2008) that includes 19 different domains. The corpus consists of approximately 500K words of contemporary American English written and spoken data drawn from the Open American National Corpus (OANC).…”

Section: Named Entity Recognitionmentioning

confidence: 99%

Efficient Benchmarking of NLP APIs using Multi-armed Bandits

Haffari

Carman

2017

Proceedings of the 15th Conference of the European Chapter of The Association for Computational Linguistics: Volume 1

View full text Add to dashboard Cite

Comparing NLP systems to select the best one for a task of interest, such as named entity recognition, is critical for practitioners and researchers. A rigorous approach involves setting up a hypothesis testing scenario using the performance of the systems on query documents. However, often the hypothesis testing approach needs to send a large number of document queries to the systems, which can be problematic. In this paper, we present an effective alternative based on the multi-armed bandit (MAB). We propose a hierarchical generative model to represent the uncertainty in the performance measures of the competing systems, to be used by Thompson Sampling to solve the resulting MAB. Experimental results on both synthetic and real data show that our approach requires significantly fewer queries compared to the standard benchmarking technique to identify the best system according to Fmeasure.

show abstract

Case Study: The Manually Annotated Sub-Corpus

Cited by 32 publications

References 15 publications

Readers vs. Writers vs. Texts: Coping with Different Perspectives of Text Understanding in Emotion Annotation

Readers vs. Writers vs. Texts: Coping with Different Perspectives of Text Understanding in Emotion Annotation

EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis

Efficient Benchmarking of NLP APIs using Multi-armed Bandits

Contact Info

Product

Resources

About