Learning and making decisions when costs and probabilities are both unknown

Zadrozny, Bianca; Elkan, Charles

doi:10.1145/502512.502540

Cited by 335 publications

(227 citation statements)

References 11 publications

Supporting

Mentioning

224

Contrasting

Unclassified

Order By: Relevance

“…The costs C P and C N in equation (7) are assumed positive numbers. Using non zero costs only for the true positive and the false positive cases is quite a reasonable structure of the cost matrix for this type of problems (Zadrozny and Elkan, 2001). The meaning of C P and C N can be expressed through some functions ( , ), 1,2…”

Section: Evaluation Of Economic Impact Of Classification Resultsmentioning

confidence: 99%

Evaluation of Classifiers for an Uneven Class Distribution Problem

Daskalaki

Kopanas

Avouris

2006

Applied Artificial Intelligence

151

View full text Add to dashboard Cite

Classification problems with uneven class distributions present several difficulties during the training as well as during the evaluation process of classifiers. A classification problem with such characteristics has resulted from a data-mining project where the objective was to predict customer insolvency. Using the dataset from the customer insolvency problem we study several alternative methodologies which have been reported to better suit the specific characteristics of this type of problems. Three different but equally important directions are examined; (a) the performance measures that should be used for problems in this domain, (b) the class distributions that should be used for the training data sets, (c) the classification algorithms to be used. The final evaluation of the resulting classifiers is based on a study of the economic impact of classification results. This study concludes to a framework that provides the "best" classifiers, identifies the performance measures that should be used as the decision criterion and suggests the "best" class distribution based on the value of the relative gain from correct classification in the positive class.This framework has been applied in the customer insolvency problem, but it is claimed that it can be applied to many similar problems with uneven class distributions that almost always require a multi-objective evaluation proces.

show abstract

Section: Evaluation Of Economic Impact Of Classification Resultsmentioning

confidence: 99%

Evaluation of Classifiers for an Uneven Class Distribution Problem

Daskalaki

Kopanas

Avouris

2006

Applied Artificial Intelligence

151

View full text Add to dashboard Cite

show abstract

“…Various methods have been proposed for finding the mapping m such as methods based on parametric estimation (Platt 1999), binning (Zadrozny and Elkan 2001) or nonparametric Isotonic Regression (Zadrozny and Elkan 2002).…”

Section: Calibration and The Pav Algorithmmentioning

confidence: 99%

PAV and the ROC convex hull

2007

View full text Add to dashboard Cite

Classifier calibration is the process of converting classifier scores into reliable probability estimates. Recently, a calibration technique based on isotonic regression has gained attention within machine learning as a flexible and effective way to calibrate classifiers. We show that, surprisingly, isotonic regression based calibration using the Pool Adjacent Violators algorithm is equivalent to the ROC convex hull method.

show abstract

“…The data pertain to a charity's periodic solicitations to potential donors; they were the data for the KDDCUP competition in 1998 (Zadrozny and Elkan, 2001;Wang et al, 2003) and are now publicly available (Blake et al, 1998 …”

Section: A Direct Marketing Campaign Case Studymentioning

confidence: 99%

Decision-Centric Active Learning of Binary-Outcome Models

Saar‐Tsechansky

Provost

2007

Information Systems Research

View full text Add to dashboard Cite

It can be expensive to acquire the data required for businesses to employ data-driven predictive modeling, for example to model consumer preferences to optimize targeting. Prior research has introduced "active learning" policies for identifying data that are particularly useful for model induction, with the goal of decreasing the statistical error for a given acquisition cost (error-centric approaches). However, predictive models are used as part of a decision-making process, and costly improvements in model accuracy do not always result in better decisions. This paper introduces a new approach for active data acquisition that targets decision-making specifically. The new decision-centric approach departs from traditional active learning by placing emphasis on acquisitions that are more likely to affect decision-making. We describe two different types of decision-centric techniques. Next, using direct-marketing data, we compare various data-acquisition techniques. We demonstrate that strategies for reducing statistical error can be wasteful in a decision-making context, and show that one decision-centric technique in particular can improve targeting decisions significantly. We also show that this method is robust in the face of decreasing quality of utility estimations, eventually converging to uniform random sampling, and that it can be extended to situations where different data acquisitions have different costs. The results suggest that businesses should consider modifying their strategies for acquiring information through normal business transactions. For example, a firm such as Amazon.com that models consumer preferences for customized marketing may accelerate learning by proactively offering recommendations-not merely to induce immediate sales, but for improving recommendations in the future.

show abstract

Learning and making decisions when costs and probabilities are both unknown

Cited by 335 publications

References 11 publications

Evaluation of Classifiers for an Uneven Class Distribution Problem

Evaluation of Classifiers for an Uneven Class Distribution Problem

PAV and the ROC convex hull

Decision-Centric Active Learning of Binary-Outcome Models

Contact Info

Product

Resources

About