DyS: A Framework for Mixture Models in Quantification

Abstract

Quantification is an expanding research topic in Machine Learning literature. While in classification we are interested in obtaining the class of individual observations, in quantification we want to estimate the total number of instances that belong to each class. This subtle difference allows the development of several algorithms that incur smaller and more consistent errors than counting the classes issued by a classifier. Among such new quantification methods, one particular family stands out due to its accuracy, simplicity, and ability to operate with imbalanced training samples: Mixture Models (MM). Despite these desirable traits, MM, as a class of algorithms, lacks a more in-depth understanding concerning the influence of internal parameters on its performance. In this paper, we generalize MM with a base framework called DyS: Distribution y-Similarity. With this framework, we perform a thorough evaluation of the most critical design decisions of MM models. For instance, we assess 15 dissimilarity functions to compare histograms with varying numbers of bins from 2 to 110 and, for the first time, make a connection between quantification accuracy and test sample size, with experiments covering 24 public benchmark datasets. We conclude that, when tuned, Topsøe is the histogram distance function that consistently leads to smaller quantification errors and, therefore, is recommended to general use, notwithstanding Hellinger Distance's popularity. To rid MM models of the dependency on a choice for the number of histogram bins, we introduce two dissimilarity functions that can operate directly on observations. We show that SORD, one of such measures, presents performance that is slightly inferior to the tuned Topsøe, while not requiring the sensible parameterization of the number of bins.

Confidence Intervals for Class Prevalences under Prior Probability Shift

“…• The methods must be Fisher consistent in the sense of Tasche (2017). This criterion excludes for instance 'classify & count' (Forman, 2008), the 'Q-measure' approach (Barranquero et al, 2013) and the distance-minimisation approaches based on the Inner Product, Kumar-Hassebrook, Cosine, and Harmonic Mean distances mentioned in Maletzke et al (2019).…”

“…On the basis of the general asymptotic efficiency of maximum likelihood estimators (Theorem 10.1.12, Casella and Berger, 2002), the maximum likelihood approach for class prevalences is a promising approach for achieving minimum confidence intervals lengths. In addition, the ML approach may be considered a representative of the class of entropy-related estimators and, as such, is closely related to the Topsøe approach which was found to perform very well in Maletzke et al (2019).…”

“…See González et al (2017) for a recent overview of the quantification problem and approaches to deal with it. The emergence of further recent papers with new proposals of prevalence estimation methods suggests that the subject is still of high interest for both researchers and practitioners (Castaño et al, 2018;Keith and O'Connor, 2018;Maletzke et al, 2019;Vaz et al, 2019).…”

“…Recent research byMaletzke et al (2019) singled out prevalence estimation methods based on minimising distances related to the Earth Mover's distance as very well and robustly performing. Earlier research byHofer (2015) already found that prevalence estimation by minimising the Earth Mover's distance worked well in the presence of general data set shift ('local drift').…”

Challenges in Benchmarking Stream Learning Algorithms with Real-world Data

“…Many options have gone untested by Goldenberg and Webb (2018) and we refer the reader to González et al (2017) for more options and to Cha and Srihari (2002) for the particularly interesting ORD, which takes the distance between different bins into account. Maletzke et al (2019) introduces SORD, a version of ORD that exempts the discretization of numeric values to compare univariate sample distributions, and can be seen as a particular and fast-to-compute case of the Earth Mover's Distance.…”