Hardness of Learning Neural Networks with Natural Weights

Daniely, Amit; Vardi, Gal

doi:10.48550/arxiv.2006.03177

Cited by 7 publications

(6 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence, the results from Klivans and Sherstov [2006] and Daniely and Shalev-Shwartz [2016] imply hardness of improperly learning depth-2 neural networks with n ǫ and ω(log(n)) hidden neurons (respectively). Daniely and Vardi [2020] showed, under the assumption that refuting a random K-SAT formula is hard, that improperly learning depth-2 neural networks is hard already if its weights are drawn from some "natural" distribution or satisfy some "natural" properties. While hardness of proper learning is implied by hardness of improper learning, there are some recent works that show hardness of properly learning depth-2 networks under more standard assumptions (cf.…”

Section: Intersections Of Halfspacesmentioning

confidence: 99%

See 1 more Smart Citation

From Local Pseudorandom Generators to Hardness of Learning

Daniely,

Vardi

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We prove hardness-of-learning results under a well-studied assumption on the existence of local pseudorandom generators. As we show, this assumption allows us to surpass the current state of the art, and prove hardness of various basic problems, with no hardness results to date.Our results include: hardness of learning shallow ReLU neural networks under the Gaussian distribution and other distributions; hardness of learning intersections of ω(1) halfspaces, DNF formulas with ω(1) terms, and ReLU networks with ω(1) hidden neurons; hardness of weakly learning deterministic finite automata under the uniform distribution; hardness of weakly learning depth-3 Boolean circuits under the uniform distribution, as well as distribution-specific hardness results for learning DNF formulas and intersections of halfspaces. We also establish lower bounds on the complexity of learning intersections of a constant number of halfspaces, and ReLU networks with a constant number of hidden neurons. Moreover, our results imply the hardness of virtually all improper PAC-learning problems (both distribution-free and distribution-specific) that were previously shown hard under other assumptions.

show abstract

Section: Intersections Of Halfspacesmentioning

confidence: 99%

“…It does not imply the hardness result fromDaniely and Vardi [2020] for learning depth-2 neural networks whose weights are drawn from some "natural" distribution.…”

mentioning

confidence: 99%

From Local Pseudorandom Generators to Hardness of Learning

Daniely,

Vardi

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…All known works take additional assumptions on the unknown weights and coefficients or do not run in polynomial-time in all the important parameters. In fact, there are a number of lower bounds, both for restricted models of computation like correlational statistical queries [GGJ + 20, DKKZ20] as well as under various average-case assumptions [DV20,DV21], that suggest that a truly polynomial-time algorithm may be impossible to achieve.…”

Section: Introductionmentioning

confidence: 99%

Efficiently Learning Any One Hidden Layer ReLU Network From Queries

Chen¹,

Klivans²,

Meka³

2021

Preprint

View full text Add to dashboard Cite

Model extraction attacks have renewed interest in the classic problem of learning neural networks from queries. This work gives the first polynomial-time algorithm for learning one hidden layer neural networks provided black-box access to the network. Formally, we show that if F is an arbitrary one hidden layer neural network with ReLU activations, there is an algorithm with query complexity and running time that is polynomial in all parameters that outputs a network F ′ achieving low square loss relative to F with respect to the Gaussian measure. While a number of works in the security literature have proposed and empirically demonstrated the effectiveness of certain algorithms for this problem, ours is the first with fully polynomial-time guarantees of efficiency for worst-case networks (in particular our algorithm succeeds in the overparameterized setting).

show abstract

“…Note in particular, that this problem includes problems of learning generative deep networks [16]. For models with hidden nodes there are many computational hardness results that are known to hold both in the worst-case [18] and in averagecase sense [9,10] even for two layer networks. Even for tree models with hidden nodes, without additional assumption, learning is as hard as learning parity with noise [23].…”

Section: Introductionmentioning

confidence: 99%

Learning to Sample from Censored Markov Random Fields

Moitra,

Mossel,

Sandon

2021

Preprint

View full text Add to dashboard Cite

We study learning Censor Markov Random Fields (abbreviated CMRFs). These are Markov Random Fields where some of the nodes are censored (not observed). We present an algorithm for learning high temperature CMRFs within o(n) transportation distance. Crucially our algorithm makes no assumption about the structure of the graph or the number or location of the observed nodes. We obtain stronger results for high girth high temperature CMRFs as well as computational lower bounds indicating that our results can not be qualitatively improved.

show abstract

Hardness of Learning Neural Networks with Natural Weights

Cited by 7 publications

References 31 publications

From Local Pseudorandom Generators to Hardness of Learning

From Local Pseudorandom Generators to Hardness of Learning

Efficiently Learning Any One Hidden Layer ReLU Network From Queries

Learning to Sample from Censored Markov Random Fields

Contact Info

Product

Resources

About