Re-assessing the “Classify and Count” Quantification Method

Moreo, Alejandro; Sebastiani, Fabrizio

doi:10.1007/978-3-030-72240-1_6

Cited by 9 publications

(17 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 1 reports the AE results of this experimentation. Our results are fairly consistent with those reported in [20,21], and seem to indicate that the strongest method of all is SLD, which obtains the best average MAE result, the best average rank, and is the best method on 13 datasets out of 30. Methods E(HDy) DS (8 times best method), PACC (4 times best method), and (to a lesser extent) ACC (2 times best method), also seem to perform very well, obtaining average ranks not statistically significantly different from the best average rank (obtained by SLD).…”

Section: Methodssupporting

confidence: 91%

“…of the posterior probabilities returned by a probabilistic classifier, and not for improving its class prevalence estimates, SLD has proven to be among the most effective quantifiers in many experiments [20,21,28].…”

Section: 13mentioning

confidence: 99%

“…The quantifiers based on Explicit Loss Minimization (ELM) represent a family of methods based on structured output learning; these quantifiers rely on classifiers that have been optimized using a quantificationoriented loss measure. QuaPy implements the following ELM-based methods, all relying on Joachims' SVM perf structured output learning algorithm [17]: 6 • SVM(Q), which attempts to minimize the 𝑄 loss, that combines a classification-oriented loss and a quantification-oriented loss, as proposed in [1]; • SVM(KLD), which attempts to minimize the Kullback-Leibler Divergence, as proposed in [9] and as first used in [10]; • SVM(NKLD), which attempts to minimize a version of the Kullback-Leibler Divergence normalized via the logistic function, as first used in [10]; • SVM(AE), which uses Absolute Error as the loss, as first used in [21]; • SVM(RAE), which uses Relative Absolute Error as the loss, as first used in [21].…”

Section: Quantifiers Based On Explicit Loss Minimizationmentioning

confidence: 99%

“…Quantification has long been regarded as a by-product of classification, which means that the model selection (i.e., hyperparameter optimization) strategies customarily adopted in quantification have simply been borrowed from classification. It has been argued in [21] that specific model selection strategies should be adopted for quantification. That is, model selection strategies for quantification should minimize quantification-oriented loss measures, and be carried out on a variety of scenarios exhibiting different degrees of distribution shift.…”

Section: Model Selectionmentioning

confidence: 99%

“…Among all ELM-based methods, we choose the one that minimizes the same loss that we adopt for evaluating the results. We do not consider other variants (SVM(Q), SVM(KLD), SVM(NKLD), SVM(RAE)) since, in recent evaluations (see, e.g.,[20,21]), they have consistently underperformed other competitors.…”

mentioning

confidence: 99%

See 4 more Smart Citations

QuaPy: A Python-Based Framework for Quantification

Moreo

Esuli

Sebastiani

2021

Proceedings of the 30th ACM International Conference on Information &Amp; Knowledge Management

Self Cite

View full text Add to dashboard Cite

QuaPy is an open-source framework for performing quantification (a.k.a. supervised prevalence estimation), written in Python. Quantification is the task of training quantifiers via supervised learning, where a quantifier is a predictor that estimates the relative frequencies (a.k.a. prevalence values) of the classes of interest in a sample of unlabelled data. While quantification can be trivially performed by applying a standard classifier to each unlabelled data item and counting how many data items have been assigned to each class, it has been shown that this "classify and count" method is outperformed by methods specifically designed for quantification. QuaPy provides implementations of a number of baseline methods and advanced quantification methods, of routines for quantificationoriented model selection, of several broadly accepted evaluation measures, and of robust evaluation protocols routinely used in the field. QuaPy also makes available datasets commonly used for testing quantifiers, and offers visualization tools for facilitating the analysis and interpretation of the results. The software is opensource and publicly available under a BSD-3 licence via GitHub 1 , and can be installed via pip 2 . CCS CONCEPTS• Computing methodologies → Learning paradigms; • Information systems → Data mining.

show abstract

Section: Methodssupporting

confidence: 91%

Section: 13mentioning

confidence: 99%

Section: Quantifiers Based On Explicit Loss Minimizationmentioning

confidence: 99%

Section: Model Selectionmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

QuaPy: A Python-Based Framework for Quantification

Moreo

Esuli

Sebastiani

2021

Proceedings of the 30th ACM International Conference on Information &Amp; Knowledge Management

Self Cite

View full text Add to dashboard Cite

show abstract

A Concise Overview of LeQua@CLEF 2022: Learning to Quantify

Esuli

Moreo

Sebastiani

et al. 2022

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

LeQua 2022 is a new lab for the evaluation of methods for "learning to quantify" in textual datasets, i.e., for training predictors of the relative frequencies of the classes of interest Y = {y1, ..., yn} in sets of unlabelled textual documents. While these predictions could be easily achieved by first classifying all documents via a text classifier and then counting the numbers of documents assigned to the classes, a growing body of literature has shown this approach to be suboptimal, and has proposed better methods. The goal of this lab is to provide a setting for the comparative evaluation of methods for learning to quantify, both in the binary setting and in the single-label multiclass setting; this is the first time that an evaluation exercise solely dedicated to quantification is organized. For both the binary setting and the single-label multiclass setting, data were provided to participants both in ready-made vector form and in raw document form. In this overview article we describe the structure of the lab, we report the results obtained by the participants in the four proposed tasks and subtasks, and we comment on the lessons that can be learned from these results.The LeQua 2022 lab (https://lequa2022.github.io/) at CLEF 2022 has a "shared task" format; it is a new lab, in two important senses:-No labs on LQ have been organized before at CLEF conferences.-Even outside the CLEF conference series, quantification has surfaced only episodically in previous shared tasks. The first such shared task was Se-mEval 2016 Task 4 "Sentiment Analysis in Twitter" [37], which comprised a binary quantification subtask and an ordinal quantification subtask (these two subtasks were offered again in the 2017 edition). Quantification also featured in the Dialogue Breakdown Detection Challenge [23], in the Dialogue Quality subtasks of the NTCIR-14 Short Text Conversation task [46], and in the NTCIR-15 Dialogue Evaluation task [47]. However, quantification was never the real focus of these tasks. For instance, the real focus of the tasks described in [37] was sentiment analysis on Twitter data, to the point that almost all participants in the quantification subtasks used the trivial "classify and count" method, and focused, instead of optimising the quantification component, on optimising the sentiment analysis component, or on picking the best-performing learner for training the classifiers used by "classify and count". Similar considerations hold for the tasks discussed in [23,46,47].

show abstract

Exploring Label Correlations for Quantification of ICD Codes

Coutinho,

Martins

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Re-assessing the “Classify and Count” Quantification Method

Cited by 9 publications

References 25 publications

QuaPy: A Python-Based Framework for Quantification

QuaPy: A Python-Based Framework for Quantification

A Concise Overview of LeQua@CLEF 2022: Learning to Quantify

Exploring Label Correlations for Quantification of ICD Codes

Contact Info

Product

Resources

About