Optimizing Text Quantifiers for Multivariate Loss Functions

Abstract

We address the problem of quantification, a supervised learning task whose goal is, given a class, to estimate the relative frequency (or prevalence) of the class in a dataset of unlabeled items. Quantification has several applications in data and text mining, such as estimating the prevalence of positive reviews in a set of reviews of a given product or estimating the prevalence of a given support issue in a dataset of transcripts of phone calls to tech support. So far, quantification has been addressed by learning a general-purpose classifier, counting the unlabeled items that have been assigned the class, and tuning the obtained counts according to some heuristics. In this article, we depart from the tradition of using general-purpose classifiers and use instead a supervised learning model for structured prediction, capable of generating classifiers directly optimized for the (multivariate and nonlinear) function used for evaluating quantification accuracy. The experiments that we have run on 5,500 binary high-dimensional datasets (averaging more than 14,000 documents each) show that this method is more accurate, more stable, and more efficient than existing state-of-the-art quantification methods. ACM Reference Format:Andrea Esuli and Fabrizio Sebastiani. 2015. Optimizing text quantifiers for multivariate loss functions.

DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis

“…It is inconclusive as to which quantification approach is better. PCC outperformed CC in (Bella et al, 2010) but underperformed CC in (Esuli and Sebastiani, 2015). Following the results from (Gao and Sebastiani, 2016), which are reported on sentiment analysis in twitter, we decided to use PCC for both of our 3 ρ is the average recall and F 1 pn the macro-average F1 score of the positive and negative classes quantification submissions.…”

A Review on Quantification Learning

“…The scenario in [Esuli and Sebastiani 2015] is better. They employ RCV1-v2, a multi-label text classification benchmark.…”

“…Thus, model assessment processes require groups of samples with sufficient variability in order to provide precise error estimates. Esuli and Sebastiani [2015] employ a quantifier based on optimizing multivariate loss functions, see Section 7.3, in the context of text classification. In their experiments, this approach outperforms other quantification algorithms in a multi-class dataset with a large number of classes (99).…”

“…The difficulty arises because a quantification error cannot be decomposed as a combination of individual errors. Esuli and Sebastiani [2015] propose to optimize the well known KL divergence (10), using SV M perf [Joachims 2005] as the learning algorithm. SV M perf is able to optimize any non-linear measure computed using the contingency table, like KL divergence.…”

Online Optimization Methods for the Quantification Problem

“…Indeed, KLD is the most frequently used measure for evaluating quantification (see e.g., [3,10,11,12]). Note that KLD is non-decomposable, i.e., the error we make by estimating p viap cannot be broken down into item-level errors.…”