Testing distributional assumptions of learning algorithms

Abstract

There are many important high dimensional function classes that have fast agnostic learning algorithms when strong assumptions on the distribution of examples can be made, such as Gaussianity or uniformity over the domain. But how can one be sufficiently confident that the data indeed satisfies the distributional assumption, so that one can trust in the output quality of the agnostic learning algorithm? We propose a model by which to systematically study the design of tester-learner pairs (A, T ), such that if the distribution on examples in the data passes the tester T then one can safely trust the output of the agnostic learner A on the data.To demonstrate the power of the model, we apply it to the classical problem of agnostically learning halfspaces under the standard Gaussian distribution and present a tester-learner pair with a combined runtime of n Õ(1/ǫ 4 ) . This qualitatively matches that of the best known ordinary agnostic learning algorithms for this task. In contrast, finite sample Gaussian distribution testers do not exist for the L 1 and EMD distance measures. A key step in the analysis is a novel characterization of concentration and anticoncentration properties of a distribution whose low-degree moments approximately match those of a Gaussian. We also use tools from polynomial approximation theory.In contrast, we show strong lower bounds on the combined run-times of tester-learner pairs for the problems of agnostically learning convex sets under the Gaussian distribution and for monotone Boolean functions under the uniform distribution over {0, 1} n . Through these lower bounds we exhibit natural problems where there is a dramatic gap between standard agnostic learning run-time and the run-time of the best tester-learner pair.

Efficient Testable Learning of Halfspaces with Adversarial Label Noise

“…Moreover, the constant-factor approximation achieved is best possible, matching the known guarantees without the testable requirement and complexity lower bounds. Prior to our work, the only known result in the testable setting, due to [RV22,GKK22], achieves error opt + ǫ with complexity d poly(1/ǫ) . A novel (and seemingly necessary) feature of our approach is that the testing components of our algorithm depend on the labels (as opposed to the label-oblivious testers of [RV22,GKK22]).…”

“…Prior to our work, the only known result in the testable setting, due to [RV22,GKK22], achieves error opt + ǫ with complexity d poly(1/ǫ) . A novel (and seemingly necessary) feature of our approach is that the testing components of our algorithm depend on the labels (as opposed to the label-oblivious testers of [RV22,GKK22]). As will be explained in the proceeding discussion, to prove Theorem 1.2 we develop a testable version of the well-known localization technique that may be of broader interest.…”

“…Ideally, one would additionally like an efficient method to test these distributional assumptions, so that: (1) if our tester accepts, then we can trust the output of the learner, and (2) it is unlikely that the tester rejects if the data satisfies the distributional assumptions. This state-of-affairs motivated the definition of a new model -introduced in [RV22] and termed testable learning -formally defined below: Definition 1.1 (Testable Learning with Adversarial Label Noise [RV22]). Fix ǫ, τ ∈ (0, 1] and let f : [0, 1] → R + .…”

“…The initial work [RV22] and the followup paper [GKK22] focused on the setting where f (t) = t (i.e., achieving optimal error of opt + ǫ). These works developed general moment-matching based algorithms that yield testable learners for a range of concept classes, including halfspaces.…”