The Extended Littlestone's Dimension for Learning with Mistakes and Abstentions

Abstract

This paper studies classification with an abstention option in the online setting. In this setting, examples arrive sequentially, the learner is given a hypothesis class H, and the goal of the learner is to either predict a label on each example or abstain, while ensuring that it does not make more than a pre-specified number of mistakes when it does predict a label.Previous work on this problem has left open two main challenges. First, not much is known about the optimality of algorithms, and in particular, about what an optimal algorithmic strategy is for any individual hypothesis class. Second, while the realizable case has been studied, the more realistic nonrealizable scenario is not well-understood. In this paper, we address both challenges. First, we provide a novel measure, called the Extended Littlestone's Dimension, which captures the number of abstentions needed to ensure a certain number of mistakes. Second, we explore the non-realizable case, and provide upper and lower bounds on the number of abstentions required by an algorithm to guarantee a specified number of mistakes.

SafePredict: A Meta-Algorithm for Machine Learning That Uses Refusals to Guarantee Correctness

“…predictor is always correct) among a set of predictors and aims to find this perfect predictor with a minimum number of refusals. Sayedi et al [31] extend this framework by allowing a fixed error budget k and characterizing the minimum number of refusals by keeping the number of errors below k. Finally, Zhang et al [32] relax the perfect predictor assumption to an l-bias assumption (i.e. predictor makes at most l errors) and allow the algorithm to refuse in order to achieve an optimal error-refuse trade-off.…”

“…where (32) follows from the sum of a geometric series and (33) follows from the definition of K * . Finally, the desired result is obtained by dividing both sides by T * .…”