The Overlap Gap Property and Approximate Message Passing Algorithms for $p$-spin models

Gamarnik, David; Jagannath, Aukosh

doi:10.48550/arxiv.1911.06943

Cited by 8 publications

(13 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our main result is a simple proof that all near-maxima of H N on [−1, 1] N are close to a corner in {±1} N , confirming the conjecture of [GJ19]. Moreover we obtain an explicit quantitative dependence, though we do not expect it to be tight.…”

Section: Introductionsupporting

confidence: 65%

“…In other words, to understand the set of near-maxima of H N on [−1, 1] N , it is in some sense sufficient to understand it on the discrete cube. Conditional on (an implication of) this result, [GJ19] prove that approximate message passing algorithms fail to approximately optimize pure p-spin models with γ p = 0 for exactly 1 value of p, over [−1, 1] N when p ≥ 4 is even. By contrast for certain other mixture functions ξ satisfying a no overlap gap condition, approximate message passing yields the only known algorithm to efficiently locate an near-maximum of H N with high probability [Mon19,AMS20].…”

Section: Introductionmentioning

confidence: 90%

See 1 more Smart Citation

Approximate Ground States of Hypercube Spin Glasses are Near Corners

Sellke¹

2020

Preprint

View full text Add to dashboard Cite

show abstract

Section: Introductionsupporting

confidence: 65%

Section: Introductionmentioning

confidence: 90%

Approximate Ground States of Hypercube Spin Glasses are Near Corners

Sellke¹

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…In particular, the recent paper [GJ19] proves that approximate message passing algorithms (of the type studied in this paper) cannot achieve a (1 − ε)-approximation of the optimum in pure p-spin Ising models, under the assumption that these exhibit an overlap gap. However [GJ19] does not characterize optimal approximation ratio, which we instead do here, as a special case of our results.…”

Section: Introductionmentioning

confidence: 95%

“…In a different direction, Gamarnik and co-authors showed in several examples that the existence of an overlap gap rules out a (1−ε)-approximation for certain classes of polynomial time algorithms [GS14, GS17, CGP + 19]. In particular, the recent paper [GJ19] proves that approximate message passing algorithms (of the type studied in this paper) cannot achieve a (1 − ε)-approximation of the optimum in pure p-spin Ising models, under the assumption that these exhibit an overlap gap. However [GJ19] does not characterize optimal approximation ratio, which we instead do here, as a special case of our results.…”

Section: Introductionmentioning

confidence: 99%

Optimization of Mean-field Spin Glasses

Alaoui¹,

Montanari²,

Sellke³

2020

Preprint

View full text Add to dashboard Cite

Mean-field spin glasses are families of random energy functions (Hamiltonians) on highdimensional product spaces. In this paper we consider the case of Ising mixed p-spin models, namely Hamiltonians H N : Σ N → R on the Hamming hypercube Σ N = {±1} N , which are defined by the property that {H N (σ)} σ∈ΣN is a centered Gaussian process with covariance E{H N (σ 1 )H N (σ 2 )} depending only on the scalar product σ 1 , σ 2 .The asymptotic value of the optimum max σ∈ΣN H N (σ) was characterized in terms of a variational principle known as the Parisi formula, first proved by Talagrand and, in a more general setting, by Panchenko. The structure of superlevel sets is extremely rich and has been studied by a number of authors. Here we ask whether a near optimal configuration σ can be computed in polynomial time.We develop a message passing algorithm whose complexity per-iteration is of the same order as the complexity of evaluating the gradient of H N , and characterize the typical energy value it achieves. When the p-spin model H N satisfies a certain no-overlap gap assumption, for any ε > 0, the algorithm outputs σ ∈ Σ N such that H N (σ) ≥ (1 − ε) max σ ′ H N (σ ′ ), with high probability. The number of iterations is bounded in N and depends uniquely on ε. More generally, regardless of whether the no-overlap gap assumption holds, the energy achieved is given by an extended variational principle, which generalizes the Parisi formula.

show abstract

“…Approximate Message Passing (AMP) algorithms are a general family of iterative algorithms that have seen widespread use in a variety of applications. First developed for compressed sensing in [DMM09,DMM10a,DMM10b], they have since been applied to many high-dimensional problems arising in statistics and machine learning, including Lasso estimation and sparse linear regression [BM11b,MAYB13], generalized linear models and phase retrieval [Ran11,SR14,SC19], robust linear regression [DM16], sparse or structured principal components analysis (PCA) [RF12, DM14, DMK + 16, MV17], group synchronization problems [PWBM18], deep learning [BS16, BSR17, MMB17], and optimization in spin glass models [Mon19,GJ19,AMS20].…”

Section: Introductionmentioning

confidence: 99%

Approximate Message Passing algorithms for rotationally invariant matrices

Zhang¹

2020

Preprint

View full text Add to dashboard Cite

Approximate Message Passing (AMP) algorithms have seen widespread use across a variety of applications. However, the precise forms for their Onsager corrections and state evolutions depend on properties of the underlying random matrix ensemble, limiting the extent to which AMP algorithms derived for white noise may be applicable to data matrices that arise in practice.In this work, we study a more general AMP algorithm for random matrices W that satisfy orthogonal rotational invariance in law, where W may have a spectral distribution that is different from the semicircle and Marcenko-Pastur laws characteristic of white noise. For a symmetric square matrix W, the forms of the Onsager correction and state evolution in this algorithm are defined by the free cumulants of its eigenvalue distribution. These forms were derived previously by Opper, C ¸akmak, and Winther using non-rigorous dynamic functional theory techniques, and we provide a rigorous proof. For rectangular matrices W that are bi-rotationally invariant in law, we derive a similar AMP algorithm that is defined by the rectangular free cumulants of the singular value distribution of W, as introduced by Benaych-Georges.To illustrate the general algorithms, we discuss an application to Principal Components Analysis with prior information for the principal components (PCs). For sufficiently large signal strength and any prior distributions of the PCs that are not mean-zero Gaussian laws, we develop a Bayes-AMP algorithm that provably achieves better estimation accuracy than the sample PCs. Contents 1. Introduction 1 2. Preliminaries on Wasserstein convergence and free probability 13 3. AMP algorithm for symmetric square matrices 16 4.

show abstract

The Overlap Gap Property and Approximate Message Passing Algorithms for $p$-spin models

Cited by 8 publications

References 16 publications

Approximate Ground States of Hypercube Spin Glasses are Near Corners

Approximate Ground States of Hypercube Spin Glasses are Near Corners

Optimization of Mean-field Spin Glasses

Approximate Message Passing algorithms for rotationally invariant matrices

Contact Info

Product

Resources

About