Covering Cubes by Random Half Cubes, with Applications to Binary Neural Networks

Kim, Jeong Han; Roche, J.R.

doi:10.1006/jcss.1997.1560

Cited by 26 publications

(25 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…0.83. Kim and Roche (1998) proved the existence of a positive constant γ such that γ ď α N ď 1´γ with high probability; see also Talagrand (1999). In this paper we show that the Krauth-Mézard conjecture α‹ is a lower bound with positive probability, under the condition that an explicit univariate function S‹pλq is maximized at λ " 0.…”

mentioning

confidence: 69%

See 1 more Smart Citation

Capacity lower bound for the Ising perceptron

Ding

Sun

2019

Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing

View full text Add to dashboard Cite

We consider the Ising perceptron with gaussian disorder, which is equivalent to the discrete cube t´1,`1u N intersected by M random half-spaces. e perceptron's capacity is α N " M N {N for the largest integer M N such that the intersection in nonempty. It is conjectured by Krauth and Mézard (1989) that the (random) ratio α N converges in probability to an explicit constant α‹ . " 0.83. Kim and Roche (1998) proved the existence of a positive constant γ such that γ ď α N ď 1´γ with high probability; see also Talagrand (1999). In this paper we show that the Krauth-Mézard conjecture α‹ is a lower bound with positive probability, under the condition that an explicit univariate function S‹pλq is maximized at λ " 0. Our proof is an application of the second moment method to a certain slice of perceptron con gurations, as selected by the so-called TAP ( ouless, Anderson, and Palmer, 1977) or AMP (approximate message passing) iteration, whose scaling limit has been characterized by Bayati andMontanari (2011) and Bolthausen (2012). For verifying the condition on S‹pλq we outline one approach, which is implemented in the current version using (nonrigorous) numerical integration packages. In a future version of this paper we intend to complete the veri cation by implementing a rigorous numerical method.Krauth and Mézard [KM89] conjectured that as N Ò 8 the ratio M N pκq{N converges to an explicit constant α ‹ pκq, which for κ " 0 is roughly 0.83. is was one of several works in the statistical physics literature analyzing various perceptron models via the "replica" or "cavity" heuristics [Gar87, Gar88, GD88, Méz89]. In particular, in the variant where J ranges not over t´1,`1u N but over the entire sphere of radius N 1{2 , the analogous threshold was computed by Gardner and Derrida [GD88]. Another common variation is to take g µ,i P t´1,`1u to be i.i.d. symmetric random signs (Bernoulli disorder). e conjectured thresholds di er between the Ising [KM89] and spherical [GD88] models, but do not depend on whether the disorder is Bernoulli or gaussian. While the classical problem is under Bernoulli disorder, we have chosen to work under gaussian disorder to remove some technical di culties. For the spherical perceptron, very sharp rigorous results have been obtained, including a proof of the predicted threshold for all nonnegative κ under Bernoulli disorder [ST03]. For the Ising perceptron, much less has been proved. One can introduce a parameter β ě 0 and de ne the associated positive-temperature partition function Z κ,β (see (7) below). e replica calculation extends to a prediction [GD88, KM89] for the limit of N´1 ln Z κ,β as N Ò 8 with M {N Ñ α. is formula has been proved to be correct at su ciently high temperature (small β) under Bernoulli disorder [Tal00]. For the original model (1), corresponding to zero temperature or β " 8, the best rigorous result to date is that for κ " 0 there exists positive γ such that γ ď M N {N ď 1´γ with high probability [KR98, Tal99] as N Ò 8 (see also [Sto13] for some work on general κ). Our m...

show abstract

mentioning

confidence: 69%

“…As noted above, the choice of gaussian disorder is a simpli cation of the Bernoulli disorder case which does not a ect the predicted threshold. We expect that the arguments of [KR98,Tal99] can be easily modi ed to give γ ď M N {N ď 1´γ with high probability in this se ing, but will not recover the value α ‹ .…”

mentioning

confidence: 96%

Capacity lower bound for the Ising perceptron

Ding

Sun

2019

Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing

View full text Add to dashboard Cite

show abstract

“…In the rigorous literature, most existing results concern the half-space model p q " 1t ě 0u. 3 For this model, it was shown by [KR98,Tal99a] that there is a small absolute constant ą 0 such that the transition must occur between and 1 ´ : that is, the partition function (1.1) is non-zero with high probability for ď , and zero with high probability for ě 1 ´ . A more recent work [DS18] (further discussed below) uses some of the methods of this paper to show, under a certain variational hypothesis, that the partition function is non-zero with non-negligible probability for ă ‹ , where ‹ is the conjectured threshold from [KM89].…”

Section: Rigorous Results On the Spherical Perceptronmentioning

confidence: 99%

“…e existing results for p q " 1t ě 0u can likely be extended to cover p q " 1t ě u for any P ℝ. 4 To be precise, the result of[DS18] is with gaussian noise (as in this paper), while the other results[KR98,Tal99a,Xu21] are for the Bernoulli noise model where , are i.i.d. symmetric random signs.…”

mentioning

confidence: 97%

Gardner formula for Ising perceptron models at small densities

Bolthausen¹,

Nakajima²,

Sun³

et al. 2021

Preprint

View full text Add to dashboard Cite

A. We consider the Ising perceptron model with spins and " pa erns, with a general activation function that is bounded above. For bounded away from zero or p q " 1t ě u, it was shown by Talagrand [Tal00, Tal11b] that for small densities , the free energy of the model converges as Ñ 8 to the replica symmetric formula conjectured in the physics literature [KM89] (see also [GD88]). We give a new proof of this result, which covers the more general class of all functions that are bounded above and satisfy a certain variance bound. e proof uses the ( rst and second) moment method conditional on the approximate message passing iterates of the model. In order to deduce our main theorem, we also prove a new concentration result for the perceptron model in the case where is not bounded away from zero.

show abstract

“…Construction of innovative methods for BNN learning has been a hot topic of research in the last decade. [6][7][8][9][10] These developments have resulted in new approaches for VLSI 3 implementations, as well as learning theory [11][12][13] and artificial intelligence. 11 The applications of BNNs in these areas have now resulted in well-developed methods.…”

Section: Introductionmentioning

confidence: 99%

Binary Neural Network Training Algorithms Based on Linear Sequential Learning

Wang

Chaudhari

2003

Int. J. Neur. Syst.

View full text Add to dashboard Cite

A key problem in Binary Neural Network learning is to decide bigger linear separable subsets. In this paper we prove some lemmas about linear separability. Based on these lemmas, we propose Multi-Core Learning (MCL) and Multi-Core Expand-and-Truncate Learning (MCETL) algorithms to construct Binary Neural Networks. We conclude that MCL and MCETL simplify the equations to compute weights and thresholds, and they result in the construction of simpler hidden layer. Examples are given to demonstrate these conclusions.

show abstract

Covering Cubes by Random Half Cubes, with Applications to Binary Neural Networks

Cited by 26 publications

References 25 publications

Capacity lower bound for the Ising perceptron

Capacity lower bound for the Ising perceptron

Gardner formula for Ising perceptron models at small densities

Binary Neural Network Training Algorithms Based on Linear Sequential Learning

Contact Info

Product

Resources

About