Sentiment Analysis Is Not Solved! Assessing and Probing Sentiment Classification

Abstract

Neural methods for SA have led to quantitative improvements over previous approaches, but these advances are not always accompanied with a thorough analysis of the qualitative differences. Therefore, it is not clear what outstanding conceptual challenges for sentiment analysis remain. In this work, we attempt to discover what challenges still prove a problem for sentiment classifiers for English and to provide a challenging dataset. We collect the subset of sentences that an (oracle) ensemble of state-of-the-art sentiment classifiers misclassify and then annotate them for 18 linguistic and paralinguistic phenomena, such as negation, sarcasm, modality, etc. 1 Finally, we provide a case study that demonstrates the usefulness of the dataset to probe the performance of a given sentiment classifier with respect to linguistic phenomena.

TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task

“…Dataset Evaluation Chen et al (2016) and Barnes et al (2019) also use model results to assess dataset difficulty for reading comprehension and sentiment analysis. Other work also explores bias in datasets and the adoption of shallow heuristics on biased datasets in natural language inference (Niven and Kao, 2019) and argument reasoning comprehension .…”

“…A similar approach was used e.g. byBarnes et al (2019).7 See the supplemental material for details on the models, training procedure, hyperparameters, and task performance.8 https://tac.nist.gov/2014/KBP/ ColdStart/guidelines/TAC_KBP_2014_Slot_ Descriptions_V1.4.pdf…”

Improving sentiment analysis with multi-task learning of negation

“…2016; Farias and Rosso 2017; Barnes et al . 2019a). Here, we have shown that explicit training via hierarchical MTL is a viable way to incorporate some of this information.…”

“…Recent research, however, challenges the idea that end-to-end learning is able to fully capture compositional effects (Verma, Kim, and Walter 2018; Barnes et al . 2019a). It is therefore worth asking whether we can help the model by providing some form of explicit training on compositional phenomena in sentiment.…”

“…A sentiment classification system that takes a naive view of negation would likely classify the sentence in (2) as negative, as negation cues often lead models to predict more negative sentiment (Wiegand et al . 2010; Barnes, Øvrelid, and Velldal 2019a). Previous research also demonstrates the need and utility for incorporating negation information in sentiment models (Wiegand et al .…”