Phase transitions in semidefinite relaxations

Abstract

Statistical inference problems arising within signal processing, data mining, and machine learning naturally give rise to hard combinatorial optimization problems. These problems become intractable when the dimensionality of the data is large, as is often the case for modern datasets. A popular idea is to construct convex relaxations of these combinatorial problems, which can be solved efficiently for large-scale datasets. Semidefinite programming (SDP) relaxations are among the most powerful methods in this family and are surprisingly well suited for a broad range of problems where data take the form of matrices or graphs. It has been observed several times that when the statistical noise is small enough, SDP relaxations correctly detect the underlying combinatorial structures. In this paper we develop asymptotic predictions for several detection thresholds, as well as for the estimation error above these thresholds. We study some classical SDP relaxations for statistical problems motivated by graph synchronization and community detection in networks. We map these optimization problems to statistical mechanics models with vector spins and use nonrigorous techniques from statistical mechanics to characterize the corresponding phase transitions. Our results clarify the effectiveness of SDP relaxations in solving high-dimensional statistical problems. Modern datasets pose new challenges to this centuries-old framework. On one hand, high-dimensional applications require the simultaneous estimation of millions of parameters. Examples span genomics (2), imaging (3), web services (4), and so on. On the other hand, the unknown object to be estimated has often a combinatorial structure: In clustering we aim at estimating a partition of the data points (5). Network analysis tasks usually require identification of a discrete subset of nodes in a graph (6, 7). Parsimonious data explanations are sought by imposing combinatorial sparsity constraints (8).There is an obvious tension between the above requirements. Although efficient algorithms are needed to estimate a large number of parameters, the maximum likelihood (ML) method often requires the solution of NP-hard (nondeterministic polynomial-time hard) combinatorial problems. A flourishing line of work addresses this conundrum by designing effective convex relaxations of these combinatorial problems (9-11).Unfortunately, the statistical properties of such convex relaxations are well understood only in a few cases [compressed sensing being the most important success story (12-14)]. In this paper we use tools from statistical mechanics to develop a precise picture of the behavior of a class of semidefinite programming relaxations. Relaxations of this type appear to be surprisingly effective in a variety of problems ranging from clustering to graph synchronization. For the sake of concreteness we will focus on three specific problems. Z 2 SynchronizationIn the general synchronization problem, we aim at estimating x 0,1 , x 0,2 , . . . , x 0,n , which are unknown elements of ...

Message‐Passing Algorithms for Synchronization Problems over Compact Groups

“…Note that the normalization is such that the signal term has spectral norm and the noise term has spectral norm 2. Special cases of this model have appeared previously: Z=2 with one frequency [22,35] and U.1/ with one frequency [35]. In fact, [22] derives AMP for the Z=2 case and proves its information-theoretic optimality.…”

“…One can check that our state evolution recurrence matches the Bayes-optimal cavity and replica predictions of [35] for Z=2 and U.1/ with one frequency. Indeed, we expect AMP to be statistically optimal in these settings (and many others too; see Section 8), and this has been proven rigorously for Z=2 [22].…”

“…We do not include the derivation of this expression, but it can be computed from belief propagation (as in [39]) or from the replica calculation (as in [35]). standard Gaussians (of the appropriate type: real, complex, or quaternionic, depending on ), and g is drawn from the Haar measure on the group.…”

“…standard Gaussians (of the appropriate type: real, complex, or quaternionic, depending on ), and g is drawn from the Haar measure on the group. We do not include the derivation of this expression, but it can be computed from belief propagation (as in [39]) or from the replica calculation (as in [35]).…”

Notes on computational-to-statistical gaps: predictions using statistical physics

“…[PWBM16b]). This model has also been studied as a Gaussian variant of community detection [DAM16] and as a model for synchronization over the group Z/2 [JMRT16].…”