Quantification-oriented learning based on reliable classifiers

Abstract

Real-world applications demand effective methods to estimate the class distribution of a sample.In many domains, this is more productive than seeking individual predictions. At a first glance, the straightforward conclusion could be that this task, recently identified as quantification, is as simple as counting the predictions of a classifier. However, due to natural distribution changes occurring in real-world problems, this solution is unsatisfactory. Moreover, current quantification models based on classifiers present the drawback of being trained with loss functions aimed at classification rather than quantification. Other recent attempts to address this issue suffer certain limitations regarding reliability, measured in terms of classification abilities. This paper presents a learning method that optimizes an alternative metric that combines simultaneously quantification and classification performance. Our proposal offers a new framework that allows the construction of binary quantifiers that are able to accurately estimate the proportion of positives, based on models with reliable classification abilities.

Why is quantification an interesting learning problem?

“…Finally, there are quantifiers based on traditional learning methods, like instance-based learning (Barranquero et al, 2013) or decision trees (Milli et al, 2013), but also using more recent approaches, like ensembles Pérez-Gallego et al (2017) and structured output learning Sebastiani, 2010, 2015;Barranquero et al, 2015). In particular (Barranquero et al, 2015) presents a method, called Q, based on building a classifier that optimizes a loss function (Qmeasure), inspired in the popular F-measure, that combines the classification and the quantification performance of the model through a parameter β.…”

“…In particular (Barranquero et al, 2015) presents a method, called Q, based on building a classifier that optimizes a loss function (Qmeasure), inspired in the popular F-measure, that combines the classification and the quantification performance of the model through a parameter β. One difficulty in implementing this idea is that not all binary learners are capable of optimizing this kind of metric, because such loss functions are not decomposable as a linear combination of the individual errors.…”

A Review on Quantification Learning

“…For this reason, most of the experiments reported in literature employ datasets taken from other problems, like classification or regression, depending on the quantification learning task studied, see for instance [Forman 2008;Bella et al 2010;Barranquero et al 2015]. In all these cases, the authors create drifted testing sets artificially.…”

“…Maybe the most popular [Tang et al 2010;Alaiz-Rodríguez et al 2008;Barranquero et al 2015] performance measure for binary (and multi-class) quantification is Kullback-Leibler Divergence, also known as discrimination information, relative entropy or normalized cross-entropy (see [Esuli and Sebastiani 2010;Forman 2008]). KL Divergence is a special case of the family of f-divergences and it can be defined for binary quantification as:…”

“…Finally, the last group of performance measures for binary quantification are those that provide normalized values [Barranquero et al 2015;Narasimhan et al 2016]. The goal is to obtain score functions that range between 1 (the optimum) and 0.…”