How Do Cultural Differences Impact the Quality of Sarcasm Annotation?: A Case Study of Indian Annotators and American Text

Abstract

Sarcasm annotation extends beyond linguistic expertise, and often involves cultural context. This paper presents our first-of-its-kind study that deals with impact of cultural differences on the quality of sarcasm annotation. For this study, we consider the case of American text and Indian annotators. For two sarcasmlabeled datasets of American tweets and discussion forum posts that have been annotated by American annotators, we obtain annotations from Indian annotators. Our Indian annotators agree with each other more than their American counterparts, and face difficulties in case of unfamiliar situations and named entities. However, these difficulties in sarcasm annotation result in statistically insignificant degradation in sarcasm classification. We also show that these disagreements between annotators can be predicted using textual properties. Although the current study is limited to two annotators and one culture pair, our paper opens up a novel direction in evaluation of the quality of sarcasm annotation, and the impact of this quality on sarcasm classification. This study forms a stepping stone towards systematic evaluation of quality of these datasets annotated by non-native annotators, and can be extended to other culture combinations.

Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)

“…Sarcasm in Text: Traditional approaches for detecting sarcasm in text have considered rule-based techniques (Veale and Hao, 2010), lexical and pragmatic features (Carvalho et al, 2009), stylistic features (Davidov et al, 2010), situational disparity (Riloff et al, 2013), incongruity (Joshi et al, 2015), or user-provided annotations such as hashtags (Liebrecht et al, 2013). Resources in this domain are collected using Twitter as a primary data source and are annotated using two main strategies: manual annotation (Riloff et al, 2013;Joshi et al, 2016a) and distant supervision through hashtags (Davidov et al, 2010;Abercrombie and Hovy, 2016). Other research leverages context to acquire shared knowledge between the speaker and the audience (Wallace et al, 2014;Bamman and Smith, 2015).…”

Agreement is overrated: A plea for correlation to assess human evaluation reliability

“…Following Craggs and Wood (2005), Sampson and Babarczy (2008), Lommel et al (2014), Joshi et al (2016) and Amidei et al (2018b) such phenomena can be explained with variability in language interpretation and quality judgement, particularly for semantic or pragmatic language aspects -such as for instance concepts such as text usability, fluency, comprehensibility etc. Human language processing and understanding are fundamental aspects of the human language.…”

Sarcasm Detection Using Multi-Channel Attention Based BLSTM On News Headline

“…The hidden layer of LSTM is also called the LSTM cell. LSTM has the capability to plot long-term dependencies by defining each memory cell with a set of gates <d, where d is the memory dimension of the hidden state of LSTM [23].…”