Simultaneous Structures in Convex Signal Recovery—Revisiting the Convex Combination of Norms

Abstract

In compressed sensing one uses known structures of otherwise unknown signals to recover them from as few linear observations as possible. The structure comes in form of some compressibility including different notions of sparsity and low rankness. In many cases convex relaxations allow to efficiently solve the inverse problems using standard convex solvers at almost-optimal sampling rates. A standard practice to account for multiple simultaneous structures in convex optimization is to add further regularizers or constraints. From the compressed sensing perspective there is then the hope to also improve the sampling rate. Unfortunately, when taking simple combinations of regularizers, this seems not to be automatically the case as it has been shown for several examples in recent works. Here, we give an overview over ideas of combining multiple structures in convex programs by taking weighted sums and weighted maximums. We discuss explicitly cases where optimal weights are used reflecting an optimal tuning of the reconstruction. In particular, we extend known lower bounds on the number of required measurements to the optimally weighted maximum by using geometric arguments. As examples, we discuss simultaneously low rank and sparse matrices and notions of matrix norms (in the "square deal" sense) for regularizing for tensor products. We state an SDP formulation for numerically estimating the statistical dimensions and find a tensor case where the lower bound is roughly met up to a factor of two.

Optimal convex lifted sparse phase retrieval and PCA with an atomic matrix norm regularizer

“…and s > 0 is a parameter that represents the sparsity (or an approximation thereof) of the vector we are interested in recovering. Similar notions of atomic norms that promote simultaneous low rank and sparsity have appeared in [25], [29]. We will show in the next section that using • * ,s as a regularizer in lifted formulations of sparse phase retrieval and PCA gives sample complexity and error bounds nearly identical to the linear regression case.…”

“…The trace regularization in our estimator encourages low rank, while the 1 regularization encourages sparsity. However, recent work [24], [25] has shown it is impossible to take advantage of both kinds of structure simultaneously with a regularizer that is merely a convex combination of the two structure-inducing regularizers; the best we can do is exploit either the low rank as in non-sparse phase retrieval, in which case we get O(p) complexity, or the s 2 -sparsity, in which case we get O(s 2 ) complexity.…”

Proof methods for robust low-rank matrix recovery

“…If in addition, a sparsity constraint is to be imposed, the problem becomes considerably more difficult. In particular, linear combinations of the convex regularizers no longer lead to sampleefficient recovery guarantees [OJF + 15] even when using optimal tuning [KSJ19]. Only under additional structural assumptions, recovery guarantees are available using an alternating minimization approach [LLJB17,GKS19].…”

Robust Sensing of Low-Rank Matrices with Non-Orthogonal Sparse Decomposition

“…The recent work [7], which builds upon ideas on generalized projections in [10] and uses a Riemannian version of Iterative Hard Thresholding to reconstruct jointly row-sparse and low-rank matrices, only provides local convergence results. The alternative approach of using optimally weighted sums or maxima of convex regularizers [19] -the only existing work apart from our predecessor paper [9] that considers nonorthogonal sparse low-rank decompositions -requires optimal tuning of the parameters under knowledge of the ground-truth. Whereas one may not hope for a general solution to these problems, the previously mentioned tractable results on nested measurements [2,10] suggest that suitable initialization methods can be found for specific applications.…”

“…) 2 leading to a bound as in (19). Unfortunately, it is not as simple to bound γ 1 (ΓK R s1,s2 − ΓK R s1,s2 , • ) 2 in an intuitive way without deriving a tight bound on the covering number of ΓK R s1,s2 in operator norm.…”