PAC-learning in the presence of one-sided classification noise

Abstract

We derive an upper and a lower bound on the sample size needed for PAClearning a concept class in the presence of one-sided classification noise. The upper bound is achieved by the strategy "Minimum One-sided Disagreement". It matches the lower bound (which holds for any learning strategy) up to a logarithmic factor. Although "Minimum One-sided Disagreement" often leads to NP-hard combinatorial problems, we show that it can be implemented quite efficiently for some simple concept classes like, for example, unions of intervals, axis-parallel rectangles, and TREE(2, n, 2, k) which is a broad subclass of 2-level decision trees. For the first class, there is an easy algorithm with time bound O(m log m). For the second-one (resp. the third-one), we design an algorithm that applies the well-known UNION-FIND data structure and has an almost quadratic time bound (resp. time bound O(n 2 m log m)).

On multiple-instance learning of halfspaces

“…Every problem known to be efficiently PAC-learnable is also known to be efficiently learnable with one-sided random classification noise, although no formal relationship is proven so far (see Simon [15] for further discussion of the one-sided random classification noise model). Thus [2] implies the efficient independent MIL-PAC-learnability of all known efficiently PAC-learnable classes.…”

Learning Instance Concepts from Multiple-Instance Data with Bags as Distributions

“…The results above show that the minimum one-sided disagreement approach (Simon 2012) can be used to learn instance concepts from MI data. Below, we show that the asymmetry of this approach is not required when learning under other performance metrics.…”

“…δ examples using a "minimum one-sided disagreement" strategy (Simon 2012). This strategy entails choosing a classifier that minimizes the number of disagreements on positivelylabeled examples while perfectly classifying all negatively-labeled examples.…”