A complete characterization of statistical query learning with applications to evolvability

Abstract

Statistical query (SQ) learning model of Kearns is a natural restriction of the PAC learning model in which a learning algorithm is allowed to obtain estimates of statistical properties of the examples but cannot see the examples themselves (Kearns, 1998 [29]). We describe a new and simple characterization of the query complexity of learning in the SQ learning model. Unlike the previously known bounds on SQ learning (Blum, et al.]) our characterization preserves the accuracy and the efficiency of learning. The preservation of accuracy implies that our characterization gives the first characterization of SQ learning in the agnostic learning framework of Haussler (1992) [23] and Kearns, Schapire and Sellie (1994) [31]. The preservation of efficiency is achieved using a new boosting technique and allows us to derive a new approach to the design of evolution algorithms in Valiant's model of evolvability (Valiant, 2009 [40]). We use this approach to demonstrate the existence of a large class of monotone evolution algorithms based on square loss performance estimation. These results differ significantly from the few known evolution algorithms and give evidence that evolvability in Valiant's model is a more versatile phenomenon than there had been previous reason to suspect.

Randomized Response Schemes, Privacy and Usefulness

“…Second, for a broad variety of classes F that actually are weakly SQ-learnable under the uniform distribution, we design a randomized response scheme that provides ε-differential privacy and, at the same time, meets a quite strong criterion of usefulness for every f ∈ F , namely allowing to infer (in probability) fromD an approximation ω of the true fraction ω of instances in D satisfying f . We would like to stress that SQ-learnability is a quite influential model with rich relations to other concepts in machine learning theory like, for instance, margin complexity or evolvability [20,7]. The results in this paper (and previous ones [14,4,10]) show that these concepts are of high relevance in the field of Differential Privacy too, so as to establish a strong connection between these two fields.…”

Privately Releasing Conjunctions and the Statistical Query Barrier

“…We show that the number of statistical queries necessary and sufficient for this task is, up to a factor of O(d), equal to the agnostic learning complexity of C (over arbitrary distributions) in Kearns' statistical query (SQ) model [Kea98]. Using an SQ lower bound for agnostically learning monotone conjunctions shown by Feldman [Fel10], this connection implies that no polynomial-time algorithm operating in the SQ-model can release even monotone conjunctions to subconstant error. Since monotone conjunction queries can be described by a submodular function, the lower bound applies to releasing submodular functions as well.…”

PAC learning and genetic programming

“…The most classical approach in computational learning theory is to study this within the approximately correct (PAC) learning framework of Valiant [14]. Recently, it has been shown that a large class of functions is evolvable, i. e. PAC learnable by an evolutionary algorithm [15,3,4]. However, the goal of these papers is to understand natural evolution from a theoretical point of view rather than giving explanations why and how evolutionary algorithms that are used in practice work.…”