PAC Learning Intersections of Halfspaces with Membership Queries

Abstract

A randomized learning algorithm POLLY is presented that efficiently learns intersections of s halfspaces in n dimensions, in time polynomial in both s and n. The learning protocol is the PAC (probably approximately correct) model of Valiant, augmented with membership queries. In particular, POLLY receives a set S of m = poly(n, s, 1/ε, 1/δ) randomly generated points from an arbitrary distribution over the unit hypercube, and is told exactly which points are contained in, and which points are not contained in, the convex polyhedron P defined by the halfspaces. POLLY may also obtain the same information about points of its own choosing. It is shown that after poly(n, s, 1/ε, 1/δ, log(1/d)) time, the probability that POLLY fails to output a collection of s halfspaces with classification error at most ε, is at most δ. Here, d is the minimum distance between the boundary of the target and those examples in S that are not lying on the boundary. The parameter log(1/d) can be bounded by the number of bits needed to encode the coefficients of the bounding hyperplanes and the coordinates of the sampled examples S. Moreover, POLLY can be extended to learn unions of k disjoint polyhedra with each polyhedron having at most s facets, in time poly(n, k, s, 1/ε, 1/δ, log(1/d), 1/γ ) where γ is the minimum distance between any two distinct polyhedra.

Learning intersections and thresholds of halfspaces

“…Building on work of Blum et al [15] and Baum [7], Kwek and Pitt [34] have given a membership query algorithm for learning the intersection of k halfspaces in R n (with respect to any probability distribution) in time polynomial in n and k:…”

Cryptographic Hardness for Learning Intersections of Halfspaces

“…A natural and important extension of the concept class of halfspaces is the concept class of intersections of halfspaces. While many efficient algorithms exist for PAC learning a single halfspace, the problem of learning the intersection of even two halfspaces remains a central challenge in computational learning theory, and a variety of efficient algorithms have been developed for natural restrictions of the problem [14,15,19,26]. Attempts to prove that the problem is hard have been met with limited success: all known hardness results for the general problem of PAC learning intersections of halfspaces apply only to the case of proper learning, where the output hypothesis must be of the same form as the unknown concept.…”

Halfspace Matrices

“…This concept class is arguably the most studied one Klivans & Sherstov 2006, 2007Kwek & Pitt 1998;Vempala 1997) in computational learning theory, with applications in areas as diverse as data mining, artificial intelligence, and computer vision. In a fundamental paper, Blum et al (1998) gave a polynomial-time algorithm for learning halfspaces in the SQ model under arbitrary distributions.…”