Learning Juntas in the Presence of Noise

Abstract

The combination of two major challenges in machine learning is investigated: dealing with large amounts of irrelevant information and learning from noisy data. It is shown that large classes of Boolean concepts that depend on a small number of variables-so-called juntas-can be learned efficiently from random examples corrupted by random attribute and classification noise.To accomplish this goal, a two-phase algorithm is presented that copes with several problems arising from the presence of noise: firstly, a suitable method for approximating Fourier coefficients in the presence of noise is applied to infer the relevant variables. Secondly, as one cannot simply read off a truth table from the examples as in the noise-free case, an alternative method to build a hypothesis is established and applied to the examples restricted to the relevant variables.In particular, for the class of monotone juntas depending on d out of n variables, the sample complexity is polynomial in log(n/δ), 2 d , γ −d a , and γ −1 b , where δ is the confidence parameter and γ a , γ b > 0 are noise parameters bounding the noise rates away from 1/2. The running time is bounded by the sample complexity times a polynomial in n.So far, all results hold for the case of uniformly distributed examples-the only case that (apart from side notes) has been studied in the literature yet. We show how to extend our methods to non-uniformly distributed examples and derive new results for monotone juntas.For the attribute noise, we have to assume that it is generated by a product distribution since otherwise fault-tolerant learning is in general impossible: we construct a noise distribution P and a concept class C such that it is impossible to learn C under P -noise.

Detecting controlling nodes of boolean regulatory networks

“…The theorem follows from applying standard Hoeffding bounds. Note that the bound above is different to [ 6 ]. If τ = 1, the number of samples required to reach a predefined probability of error is smaller by a factor 4.…”

“…In fact, one can readily apply methods stemming from the area of PAC (probably approximately correct) learning theory [ 5 ], as the network identification problem can be reduced to the problem of learning Boolean juntas , i.e., Boolean functions that depend b only on a small number of their arguments. This problem was studied by Arpe and Reischuk [ 6 ] extending earlier work of Mossel et al [ 7 , 8 ].…”

“…Hence, if is larger than 2 -k , the variables corresponding to U are classified as essential. The algorithm was given by [ 6 ], but they used 2 - d -1 as threshold (see Line 8).…”

“…This directly follows from the different threshold used here. If τ > 1, it was claimed in [ 6 ] that n τ can be replaced by n . But simulation results of the authors (not shown) contradict this result; hence, we rely here on the weaker result shown in Theorem 2.…”

“…The application of spectral learning algorithms leads to both better time and sample complexity for the detection of controlling nodes compared with exhaustive search. In particular, a slight improvement in the algorithm given in [ 6 ] is presented, for which analytical bounds on the number of samples needed to find the controlling nodes are derived. It is notable that for the class of 1-low networks, the time complexity of the resulting algorithms is roughly n 2 .…”