Consistent algorithms for multiclass classification with an abstain option

Ramaswamy, Harish G.; Tewari, Ambuj; Agarwal, Shivani

doi:10.1214/17-ejs1388

Cited by 31 publications

(36 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Classifiers that output possibly more than one label are known as set-valued classifiers (Grycko, 1993) or non-deterministic classifiers (del Coz et al, 2009). In another related framework called classification with reject option (Chow, 1970;Herbei and Wegkamp, 2006;Bartlett and Wegkamp, 2008;Yuan and Wegkamp, 2010;Ramaswamy et al, 2015), a classifier may reject to output a definitive class label if the uncertainty is high. Set-valued classification contains this framework as a special case, as one can view the "reject to classify" option as outputting the entire set of possible labels.…”

Section: Related Workmentioning

confidence: 99%

Least Ambiguous Set-Valued Classifiers With Bounded Error Levels

Sadinle

Lei

Wasserman

2018

Journal of the American Statistical Association

111

View full text Add to dashboard Cite

In most classification tasks there are observations that are ambiguous and therefore difficult to correctly label. Set-valued classification allows the classifiers to output a set of plausible labels rather than a single label, thereby giving a more appropriate and informative treatment to the labeling of ambiguous instances. We introduce a framework for multiclass set-valued classification, where the classifiers guarantee user-defined levels of coverage or confidence (the probability that the true label is contained in the set) while minimizing the ambiguity (the expected size of the output). We first derive oracle classifiers assuming the true distribution to be known. We show that the oracle classifiers are obtained from level sets of the functions that define the conditional probability of each class. Then we develop estimators with good asymptotic and finite sample properties. The proposed classifiers build on and refine many existing single-label classifiers. The optimal classifier can sometimes output the empty set. We provide two solutions to fix this issue that are suitable for various practical needs.

show abstract

Section: Related Workmentioning

confidence: 99%

Least Ambiguous Set-Valued Classifiers With Bounded Error Levels

Sadinle

Lei

Wasserman

2018

Journal of the American Statistical Association

111

View full text Add to dashboard Cite

show abstract

“…There is a rapidly increasing line of work devoted to designing classifiers that are able to defer decisions (Bartlett and Wegkamp 2008;Cortes, DeSalvo, and Mohri 2016;Geifman, Uziel, and El-Yaniv 2018;Geifman and El-Yaniv 2019;Raghu et al 2019b;Ramaswamy et al 2018;Thulasidasan et al 2019;Liu et al 2019). Here, the classifiers learn to defer either by considering the defer action as an additional label value or by training an independent classifier to decide about deferred decisions.…”

Section: Introductionmentioning

confidence: 99%

Regression under Human Assistance

Koley

Ganguly

et al. 2020

AAAI

View full text Add to dashboard Cite

Decisions are increasingly taken by both humans and machine learning models. However, machine learning models are currently trained for full automation—they are not aware that some of the decisions may still be taken by humans. In this paper, we take a first step towards the development of machine learning models that are optimized to operate under different automation levels. More specifically, we first introduce the problem of ridge regression under human assistance and show that it is NP-hard. Then, we derive an alternative representation of the corresponding objective function as a difference of nondecreasing submodular functions. Building on this representation, we further show that the objective is nondecreasing and satisfies α-submodularity, a recently introduced notion of approximate submodularity. These properties allow a simple and efficient greedy algorithm to enjoy approximation guarantees at solving the problem. Experiments on synthetic and real-world data from two important applications—medical diagnosis and content moderation—demonstrate that the greedy algorithm beats several competitive baselines.

show abstract

“…One type of procedure allows a rejection option, that is, if a future object falls into a 'rejection' region, then no classification is made for the object. Such a procedure aims to construct a suitable rejection region to minimize a pre-specified risk; see, e.g., [5][6][7][8][9] and the references therein. Non-deterministic classifiers are proposed in [10], which allow a future object to be classified possibly into several classes.…”

Section: Introductionmentioning

confidence: 99%

Confidence Sets for Statistical Classification

Liu

Bretz

Srimaneekarn

et al. 2019

Stats

View full text Add to dashboard Cite

Classification has applications in a wide range of fields including medicine, engineering, computer science and social sciences among others. In statistical terms, classification is inference about the unknown parameters, i.e., the true classes of future objects. Hence, various standard statistical approaches can be used, such as point estimators, confidence sets and decision theoretic approaches. For example, a classifier that classifies a future object as belonging to only one of several known classes is a point estimator. The purpose of this paper is to propose a confidence-set-based classifier that classifies a future object into a single class only when there is enough evidence to warrant this, and into several classes otherwise. By allowing classification of an object into possibly more than one class, this classifier guarantees a pre-specified proportion of correct classification among all future objects. An example is provided to illustrate the method, and a simulation study is included to highlight the desirable feature of the method.

show abstract

Consistent algorithms for multiclass classification with an abstain option

Cited by 31 publications

References 15 publications

Least Ambiguous Set-Valued Classifiers With Bounded Error Levels

Least Ambiguous Set-Valued Classifiers With Bounded Error Levels

Regression under Human Assistance

Confidence Sets for Statistical Classification

Contact Info

Product

Resources

About