Improved Lower Bounds for Learning Intersections of Halfspaces

Abstract

Recent work of Klivans, Stavropoulos, and Vasilyan initiated the study of testable learning with distribution shift (TDS learning), where a learner is given labeled samples from training distribution D, unlabeled samples from test distribution D ′ , and the goal is to output a classifier with low error on D ′ whenever the training samples pass a corresponding test. Their model deviates from all prior work in that no assumptions are made on D ′ . Instead, the test must accept (with high probability) when the marginals of the training and test distributions are equal.Here we focus on the fundamental case of intersections of halfspaces with respect to Gaussian training distributions and prove a variety of new upper bounds including a 2 (k/ǫ) O( 1) poly(d)-time algorithm for TDS learning intersections of k homogeneous halfspaces to accuracy ǫ (prior work achieved d (k/ǫ) O(1) ). We work under the mild assumption that the Gaussian training distribution contains at least an ǫ fraction of both positive and negative examples (ǫ-balanced). We also prove the first set of SQ lower-bounds for any TDS learning problem and show (1) the ǫ-balanced assumption is necessary for poly(d, 1/ǫ)-time TDS learning for a single halfspace and (2) a d Ω(log 1/ǫ) lower bound for the intersection of two general halfspaces, even with the ǫ-balanced assumption.Our techniques significantly expand the toolkit for TDS learning. We use dimension reduction and coverings to give efficient algorithms for computing a localized version of discrepancy distance, a key metric from the domain adaptation literature.

Cryptographic Hardness for Learning Intersections of Halfspaces

“…Until recently, the problem was known to be hard only for proper learning: if the learner's output hypothesis must be from a restricted class of functions (e.g., intersections of halfspaces), then the learning problem is NP-hard with respect to randomized reductions [4,2]. Klivans and Sherstov [16] have since obtained a 2 Ω( √ n) lower bound on the sample complexity of learning intersections of √ n halfspaces in the statistical query (SQ) model, an important restriction of the PAC model. Since the SQ model is stronger than PAC, the lower bounds in [16] do not not imply hardness in the PAC model, the subject of this paper.…”

“…Klivans and Sherstov [16] have since obtained a 2 Ω( √ n) lower bound on the sample complexity of learning intersections of √ n halfspaces in the statistical query (SQ) model, an important restriction of the PAC model. Since the SQ model is stronger than PAC, the lower bounds in [16] do not not imply hardness in the PAC model, the subject of this paper. We are not aware of any other results on the difficulty of learning intersections of halfspaces.…”

“…Intersections of a polynomial number of halfspaces, however, cannot compute the decryption functions of the cryptosystems we use. In fact, the decryption functions in question contain PARITY as a subfunction (see Section 7), which cannot be computed by intersections of a polynomial number of any unate functions [16]. Furthermore, the decryption functions for Regev's cryptosystems perform a division or an iterated addition, which require threshold depth 3 and 2, respectively [27,24].…”

Evolvability from learning algorithms

“…without relying on any computational hardness assumptions) [16,4]. This result is a special case of lower bounds based on the SQ dimension which was introduced by Blum et al [4] and studied in a number of subsequent works [5,6,26,18,22]. Limitations on learnability in the SQ model imply limitations on evolutionary algorithms.…”

New Results for Learning Noisy Parities and Halfspaces

“…There are numerous negative results known for proper learning of such concepts [38,2], and for learning in the Statistical Query model [32]. Based on certain cryptographic assumptions, Kearns and Valiant showed that constant depth threshold circuits cannot be learned over a certain distribution using any representation [24].…”