NRU-HSE at SemEval-2016 Task 4: Comparative Analysis of Two Iterative Methods Using Quantification Library

Abstract

In many areas, such as social science, politics or market research, people need to track sentiment and their changes over time. For sentiment analysis in this field it is more important to correctly estimate proportions of each sentiment expressed in the set of documents (quantification task) than to accurately estimate sentiment of a particular document (classification). Basically, our study was aimed to analyze the effectiveness of two iterative quantification techniques and to compare their effectiveness with baseline methods. All the techniques are evaluated using a set of synthesized data and the SemEval-2016 Task4 dataset. We made the quantification methods from this paper available as a Python open source library. The results of comparison and possible limitations of the quantification techniques are discussed.

Fisher consistency for prior probability shift

“…Relative error behaves similar to Kullback-Leibler distance used by Karpov et al (2016), but is defined also for q = 0 and q = 1 and, moreover, has a more intuitive interpretation.…”

“…A variety of other approaches have been and are being studied in the literature (see the discussion in Hofer, 2015, for a recent overview). The selection of approaches to be discussed in this paper was driven by findings of Xue and Weiss ( 2009) and more recently Karpov et al (2016). According to that research, for the estimation of binary class prevalences the CDE-Iterate approach seems to perform equally well or even stronger than ACC which by itself was found to outperform the popular EM-algorithm.…”

“…Xue and Weiss (2009) proposed 'CDE-Iterate' (CDE for class distribution estimation) which is appropriately summarised by Karpov et al (2016) as follows: "The main idea of this method is to retrain a classifier at each iteration, where the iterations progressively improve the quantification accuracy of performing the 'classify and count' method via the generated cost-sensitive classifiers." Xue and Weiss motivated the CDE-Iterate algorithm as kind of an equivalent of the EM-algorithm, with the training set feature-conditional class probabilities replaced by Bayes classifiers (or approximation of the classifiers) learnt on the training set.…”

“…Modelled class prevalences. The setting is broadly the same as for the artificial dataset in Karpov et al (2016). For each model, we consider a training set with class probabilities 50%, combined with test sets with class 0 probabilities 1%, 5%, 10%, 30%, 50%, 70%, 90%, 95% and 99%.…”

NRU-HSE at SemEval-2017 Task 4: Tweet Quantification Using Deep Learning Architecture

“…We participated in D and E subtasks of the tweet sentiment quantification competition SemEval-2017 Task 4. To solve them we used a quantification method, which showed good accuracy last year (Karpov et al, 2016) and deep learning architecture mentioned in literature for text classification task.…”