A Theory of PAC Learnability of Partial Concept Classes

Abstract

We extend the classical theory of PAC learning in a way which allows to model a rich variety of practical learning tasks where the data satisfy special properties that ease the learning process. For example, tasks where the distance of the data from the decision boundary is bounded away from zero, or tasks where the data lie on a lower dimensional surface. The basic and simple idea is to consider partial concepts: these are functions that can be undefined on certain parts of the space. When learning a partial concept, we assume that the source distribution is supported only on points where the partial concept is defined.This way, one can naturally express assumptions on the data such as lying on a lower dimensional surface, or that it satisfies margin conditions. In contrast, it is not at all clear that such assumptions can be expressed by the traditional PAC theory using learnable total concept classes, and in fact we exhibit easy-to-learn partial concept classes which provably cannot be captured by the traditional PAC theory. This also resolves, in a strong negative sense, a question posed by Attias, Kontorovich, and Mansour (2019).We characterize PAC learnability of partial concept classes and reveal an algorithmic landscape which is fundamentally different than the classical one. For example, in the classical PAC model, learning boils down to Empirical Risk Minimization (ERM). This basic principle follows from

Best lower bound on the probability of a binomial exceeding its expectation

“…Such results have applications to the so-called probablyapproximately-correct (PAC) models of machine learning; concerning PAC models, see e.g. [12,6,8,1].…”

“…(Concerning the strictness of the inequality P(X E X) > 1/4 in (4), here one may recall that the inequality P(X > E X) 1/4 in ( 1) is strict unless n = 2 and p = 1/2 -in which latter case condition (5) fails to hold.) (ii) Instead of the probability P(X E X) in (4), we have the (possibly) smaller probability P(X > E X) in (1). Improvement (i) and the optimality of the constant factor c are illustrated in Figure 1, showing the graphs • { p, P(X n,p > np) : 1/n p < 1} (solid)…”

“…Case 5: Then n > np n − 1, p 1 − 1/n, and hence P(X > np) = P(X = n) = p n (1 − 1/n) n , and (1−1/n) n is increasing in n 2, from (1−1/2) 2 = 1/4. So, P(X > np) > 1 4 in Case 5 -except when n = 2 and p = 1/2, in which case P(X > np) = 1 4 . This completes the second proof of Theorem 1.…”

“…Also, the conditions np 2 and nq 2 imply n = np + nq 4. So, in Case 1, P(X > np) 1 2 − max(ε * (4), ε * (89), ε * (90)) > 0.25587 > 1/4. Case 2: Then, by (9), P(X > np) > 1 4 .…”

Three-way Learnability: A Learning Theoretic Perspective on Three-way Decision

“…More recently, the application of orthopairs in CLT has been studied in the setting of adversarial machine learning [22], as well as to characterize the generalization 1 Here realizability means that ∃h ∈ H s.t. L D (h) = 0. capacity of hypothesis classes under generative assumptions [23]. We note, however, that even though the above mentioned work and the framework we study in this article rely on the representation formalism of orthopairs, the aims of these three frameworks are essentially orthogonal, also in terms of the mathematical techniques adopted: Indeed, while the three-way learning framework we study relies on a generalization of the ERM paradigm, the frameworks studied in [23], [22] rely on a transductive learning approach.…”