Variable-constraint classification and quantification of radiology reports under the ACR Index

Lecture Notes in Computer Science

2014

Self Cite

What Is Quantification?In recent years it has been pointed out that, in a number of applications involving classification, the final goal is not determining which class (or classes) individual unlabelled data items belong to, but determining the prevalence (or "relative frequency") of each class in the unlabelled data. The latter task has come to be known as quantification [1, 3, 5-10, 15, 18, 19].Although what we are going to discuss here applies to any type of data, in this tutorial we will mostly be interested in text quantification, i.e., quantification when the data items are textual documents. To see the importance of text quantification, let us examine the task of classifying textual answers returned to open-ended questions in questionnaires [4,11,12], and let us discuss two important such scenarios.In the first scenario, a telecommunications company asks its current customers the question "How satisfied are you with our mobile phone services?", and wants to classify the resulting textual answers according to whether they belong or not to class MayDefectToCompetition. (Membership in this class indicates that the customer is so unhappy with the company's services that she is probably considering a switch to another company.) The company is likely interested in accurately classifying each individual customer, since it may want to call each customer that is assigned the class and offer her improved conditions.In the second scenario, a market research agency asks respondents the question "What do you think of the recent ad campaign for product X?", and wants to classify the resulting textual answers according to whether they belong to the class LovedTheCampaign. Here, the agency is likely not interested in whether a specific individual belongs to the class LovedTheCampaign, but is likely interested in knowing how many respondents belong to it, i.e., in knowing the prevalence of the class.In sum, while in the first scenario classification is the goal, in the second scenario the real goal is quantification. Essentially, quantification is classification evaluated at the aggregate (rather than at the individual) level. Other scenarios in which quantification is the goal may be, e.g., predicting election results by estimating the prevalence of blog posts (or tweets) supporting a given candidate or party [13], or planning the amount of human resources to allocate to different types of issues in a customer support center by estimating the prevalence of M. de Rijke et al.

Section: What Is Quantification?mentioning

confidence: 99%

Text Quantification

Lecture Notes in Computer Science

2014

Self Cite

“…Gonzalez-Castro et al [2013] andSanchez et al [2008] use quantification for establishing the prevalence of damaged sperm cells in a given sample for veterinary applications. Baccianella et al [2013] classify radiology reports with the aim of estimating the prevalence of different pathologies. Tang et al [2010] focus on network quantification problems, that is, problems in which the goal is to estimate class prevalence among a population of nodes in a network.…”

Section: Applications Of Quantificationmentioning

confidence: 99%

“…Other scenarios in which quantification is the goal may be, for example, predicting election results by estimating the prevalence of blog posts (or tweets) supporting a given candidate or party [Hopkins and King 2010], planning the amount of human resources to allocate to different types of issues in a customer support center by estimating the prevalence of customer calls related to a given issue [Forman 2005], supporting epidemiological research by estimating the prevalence of medical reports in which a specific pathology is diagnosed [Baccianella et al 2013].…”

Section: Introductionmentioning

confidence: 99%

Optimizing Text Quantifiers for Multivariate Loss Functions

Esuli

ACM Trans. Knowl. Discov. Data

2015

Self Cite

We address the problem of quantification, a supervised learning task whose goal is, given a class, to estimate the relative frequency (or prevalence) of the class in a dataset of unlabeled items. Quantification has several applications in data and text mining, such as estimating the prevalence of positive reviews in a set of reviews of a given product or estimating the prevalence of a given support issue in a dataset of transcripts of phone calls to tech support. So far, quantification has been addressed by learning a general-purpose classifier, counting the unlabeled items that have been assigned the class, and tuning the obtained counts according to some heuristics. In this article, we depart from the tradition of using general-purpose classifiers and use instead a supervised learning model for structured prediction, capable of generating classifiers directly optimized for the (multivariate and nonlinear) function used for evaluating quantification accuracy. The experiments that we have run on 5,500 binary high-dimensional datasets (averaging more than 14,000 documents each) show that this method is more accurate, more stable, and more efficient than existing state-of-the-art quantification methods. ACM Reference Format:Andrea Esuli and Fabrizio Sebastiani. 2015. Optimizing text quantifiers for multivariate loss functions.

“…In the literature, AE also goes by the name of Variational Distance (Csiszár and Shields 2004, §4), (Lin 1991;Zhang and Zhou 2010), or Percentage Discrepancy (Esuli and Sebastiani 2010;Baccianella et al 2013). Also, if viewed as a generic function of dissimilarity between vectors (and not just probability distributions), AE is nothing else than the well-known "city-block distance" normalized by the number of classes.…”

Section: C|mentioning

confidence: 99%

Evaluation measures for quantification: an axiomatic approach

2019

Inf Retrieval J

Self Cite

Quantification is the task of estimating, given a set σ of unlabelled items and a set of classes C = {c 1 , . . . , c |C| }, the prevalence (or "relative frequency") in σ of each class c i ∈ C. While quantification may in principle be solved by classifying each item in σ and counting how many such items have been labelled with c i , it has long been shown that this "classify and count" (CC) method yields suboptimal quantification accuracy. As a result, quantification is no longer considered a mere byproduct of classification, and has evolved as a task of its own. While the scientific community has devoted a lot of attention to devising more accurate quantification methods, it has not devoted much to discussing what properties an evaluation measure for quantification (EMQ) should enjoy, and which EMQs should be adopted as a result. This paper lies down a number of interesting properties that an EMQ may or may not enjoy, discusses if (and when) each of these properties is desirable, surveys the EMQs that have been used so far, and discusses whether they enjoy or not the above properties. As a result of this investigation, some of the EMQs that have been used in the literature turn out to be severely unfit, while others emerge as closer to what the quantification community actually needs. However, a significant result is that no existing EMQ satisfies all the properties identified as desirable, thus indicating that more research is needed in order to identify (or synthesize) a truly adequate EMQ.