Finding a large submatrix of a Gaussian random matrix

Abstract

We consider the problem of finding a k × k submatrix of an n × n matrix with i.i.d. standard Gaussian entries, which has a large average entry. It was shown in [BDN12] using non-constructive methods that the largest average value of a k × k submatrix is 2(1 + o(1)) log n/k with high probability (w.h.p.) when k = O(log n/ log log n). In the same paper an evidence was provided that a natural greedy algorithm called Largest Average Submatrix (LAS) for a constant k should produce a matrix with average entry at most (1 + o(1)) 2 log n/k, namely approximately √ 2 smaller, though no formal proof of this fact was provided.In this paper we show that the matrix produced by the LAS algorithm is indeed (1+o(1)) 2 log n/k w.h.p. when k is constant and n grows. Then by drawing an analogy with the problem of finding cliques in random graphs, we propose a simple greedy algorithm which produces a k × k matrix with asymptotically the same average value (1 + o(1)) 2 log n/k w.h.p., for k = o(log n). Since the greedy algorithm is the best known algorithm for finding cliques in random graphs, it is tempting to believe that beating the factor √ 2 performance gap suffered by both algorithms might be very challenging. Surprisingly, we show the existence of a very simple algorithm which produces a k × k matrix with average value (1 + o k (1))(4/3) 2 log n/k for in fact k = o(n).To get an insight into the algorithmic hardness of this problem, and motivated by methods originating in the theory of spin glasses, we conduct the so-called expected overlap analysis of matrices with average value asymptotically (1 + o(1))α 2 log n/k for a fixed value α ∈ [1,√ 2]. The overlap corresponds to the number of common rows and common columns for pairs of matrices achieving this value (see the paper for details). We discover numerically an intriguing phase transition at α * 5 √ 2/(3 √ 3) ≈ 1.3608.. ∈ [4/3, √ 2]: when α < α * the space of overlaps is a continuous subset of [0, 1] 2 , whereas α = α * marks the onset of discontinuity, and as a result the model exhibits the Overlap Gap Property (OGP) when α > α * , appropriately defined. We conjecture that OGP observed for α > α * also marks the onset of the algorithmic hardness -no polynomial time algorithm exists for finding matrices with average value at least (1 + o(1))α 2 log n/k, when α > α * and k is a growing function of n.

Detection of Cyber-Physical Faults and Intrusions from Physical Correlations

“…[3], and consists in consecutive updates of k rows and k columns, starting from a random k × k submatrix and repeating the updates until a guaranteed convergence to a local maximum, meaning that the resulting submatrix can not be improved by changing only its column or row set. A recently introduced improved version of this algorithm, analysed in [16] and named Iterative Greedy Procedure (IGP) follows a simple greedy scheme: starting by one randomly chosen row, we add the best columns and rows sequentially until a k × k submatrix is recovered. This algorithm outputs a provably better results, at least in the case of large Gaussian random matrices.…”

The Overlap Gap Property in Principal Submatrix Recovery

“…In other words, the one-dimensional set of distances between the near optimal states is disconnected. This property has been established for various problems arising from theoretical computer science and combinatorial optimization, for instance random constraint satisfaction [2,33,55], Max Independent Set [32,66], and a maxcut problem on hypergraphs [24]. Further, OGP has been shown to act as a barrier to the success of a family of "local algorithms" on sparse random graphs [26,24,32,33].…”

“…This problem has natural connections to the planted clique problem [5], sparse PCA [6], biclustering [17,32,69], and community detection [1,56,59]. All these problems are expected to exhibit a statistical-computational gap-there are regimes where optimal statistical performance might be impossible to achieve using computationally feasible statistical procedures.…”

“…This property has been established for various problems arising from theoretical computer science and combinatorial optimization, for instance random constraint satisfaction [2,33,55], Max Independent Set [32,66], and a maxcut problem on hypergraphs [24]. Further, OGP has been shown to act as a barrier to the success of a family of "local algorithms" on sparse random graphs [26,24,32,33]. This perspective has been introduced to the study of inference problems arising in Statistics and Machine Learning [13,32,34,35,36] in recent works by the first two authors which has yielded new insights into algorithmic barriers in these problems.…”

“…Further, OGP has been shown to act as a barrier to the success of a family of "local algorithms" on sparse random graphs [26,24,32,33]. This perspective has been introduced to the study of inference problems arising in Statistics and Machine Learning [13,32,34,35,36] in recent works by the first two authors which has yielded new insights into algorithmic barriers in these problems. As an aside, we note that exciting new developments in the study of mean field spin glasses [3,57,70], establish that in certain problems without OGP, it is actually possible to obtain approximate optimizers using appropriately designed polynomial time algorithms.…”