The Global Optimization Geometry of Low-Rank Matrix Optimization

Zhu, Zhihui; Li, Qiuwei; Tang, Gongguo; Wakin, Michael B.

doi:10.48550/arxiv.1703.01256

Cited by 21 publications

(35 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The proof of Lemma 3.2 is given in Appendix D. As is shown in existing literature [21,22,23], the Gaussian linear operator A : R N ×N → R M introduced at the beginning of Section 3.1 satisfies the RIP condition (3.4) with high probability if M ≥ C(r + k)N 1 δ 2 r+k for some numerical constant C. Therefore, we can conclude that the three statements in Theorem 2.1 hold for the empirical risk (3.2) and population risk (3.3) as long as M is large enough. Some similar bounds for the sample complexity M under different settings can also be found in papers [7,4]. Note that the particular choice of l can guarantee that grad f (U) F is large outside of B(l), which is also proved in Appendix D. Together with Theorem 2.1, we prove a globally benign landscape for the empirical risk.…”

Section: Verifying Assumptions 21 22 and 23supporting

confidence: 70%

“…The landscape of the above population risk has been studied in the general R N ×k space with k = r in [7]. The landscape of its variants, such as the asymmetric version with or without a balanced term, has also been studied in [4,18]. It is well known that there exists an ambiguity in the solution of (3.2) due to the fact that UU = UQQ U holds for any orthogonal matrix Q ∈ R k×k .…”

Section: Matrix Sensingmentioning

confidence: 99%

“…Denote B(l) as a compact and connected subset in a general manifold M with N and l being its parameters. 4 Let g : B(l) → R be a C 2 function satisfying λ min (hess g(x)) > η in D with D {x ∈ B(l) : 4 The subset B(l) can vary in different applications. For example, we define B(l) {U ∈ R N ×k * : UU F ≤ l} in matrix sensing and B(l) {x ∈ R N : x 2 ≤ l} in phase retrieval.…”

Section: A Proof Of Theorem 21mentioning

confidence: 99%

“…Recently, the landscapes of empirical and population risk have been extensively studied in many fields of science and engineering, including machine learning and signal processing. In particular, the local or global geometry has been characterized in a wide variety of convex and non-convex problems, such as matrix sensing [3,4], matrix completion [5,6], low-rank matrix factorization [7,8], phase retrieval [9,10], blind deconvolution [11,12], tensor decomposition [13,14], and so on. In this work, we focus on analyzing global geometry, which requires understanding not only regions near critical points but also the landscape away from these points.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

The Landscape of Non-convex Empirical Risk with Degenerate Population Risk

Li,

Tang,

Wakin

2019

Preprint

Self Cite

View full text Add to dashboard Cite

The landscape of empirical risk has been widely studied in a series of machine learning problems, including low-rank matrix factorization, matrix sensing, matrix completion, and phase retrieval. In this work, we focus on the situation where the corresponding population risk is a degenerate non-convex loss function, namely, the Hessian of the population risk can have zero eigenvalues. Instead of analyzing the non-convex empirical risk directly, we first study the landscape of the corresponding population risk, which is usually easier to characterize, and then build a connection between the landscape of the empirical risk and its population risk. In particular, we establish a correspondence between the critical points of the empirical risk and its population risk without the strongly Morse assumption, which is required in existing literature but not satisfied in degenerate scenarios. We also apply the theory to matrix sensing and phase retrieval to demonstrate how to infer the landscape of empirical risk from that of the corresponding population risk.

show abstract

Section: Verifying Assumptions 21 22 and 23supporting

confidence: 70%

Section: Matrix Sensingmentioning

confidence: 99%

Section: A Proof Of Theorem 21mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

The Landscape of Non-convex Empirical Risk with Degenerate Population Risk

Li,

Tang,

Wakin

2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…On the other hand, first-order method such as projected gradient descent (PGD) for solving (2) suffers from very slow convergence. As a proof of concept, we show in Figure 1 the convergence of PGD for solving symmetric NMF and as a comparison, the convergence of gradient descent (GD) for solving a matrix factorization (MF) (i.e., (2) without the nonnegative constraint) which is proved to admit linear convergence [13,14]. This phenomenon also appears in nonsymmetric NMF and is the main motivation to have many efficient algorithms such as ANLS and HALS.…”

Section: Introductionmentioning

confidence: 96%

Dropping Symmetry for Fast Symmetric Nonnegative Matrix Factorization

Zhu,

Li,

Liu

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

Symmetric nonnegative matrix factorization (NMF)-a special but important class of the general NMF-is demonstrated to be useful for data analysis and in particular for various clustering tasks. Unfortunately, designing fast algorithms for Symmetric NMF is not as easy as for the nonsymmetric counterpart, the later admitting the splitting property that allows efficient alternating-type algorithms.To overcome this issue, we transfer the symmetric NMF to a nonsymmetric one, then we can adopt the idea from the state-of-the-art algorithms for nonsymmetric NMF to design fast algorithms solving symmetric NMF. We rigorously establish that solving nonsymmetric reformulation returns a solution for symmetric NMF and then apply fast alternating based algorithms for the corresponding reformulated problem. Furthermore, we show these fast algorithms admit strong convergence guarantee in the sense that the generated sequence is convergent at least at a sublinear rate and it converges globally to a critical point of the symmetric NMF. We conduct experiments on both synthetic data and image clustering to support our result.

show abstract

The Global Optimization Geometry of Shallow Linear Neural Networks

et al. 2019

Self Cite

View full text Add to dashboard Cite

We examine the squared error loss landscape of shallow linear neural networks. We show-with significantly milder assumptions than previous worksthat the corresponding optimization problems have benign geometric properties: there are no spurious local minima and the Hessian at every saddle point has at least one negative eigenvalue. This means that at every saddle point there is a directional negative curvature which algorithms can utilize to further decrease the objective value. These geometric properties imply that many local search algorithms (such as the gradient descent which is widely utilized for training neural networks) can provably solve the training problem with global convergence. 1From an optimization perspective, non-strict saddle points and local minima have similar first-/second-order information and it is hard for first-/second-order methods (like gradient descent) to distinguish between them.

show abstract

The Global Optimization Geometry of Low-Rank Matrix Optimization

Cited by 21 publications

References 24 publications

The Landscape of Non-convex Empirical Risk with Degenerate Population Risk

The Landscape of Non-convex Empirical Risk with Degenerate Population Risk

Dropping Symmetry for Fast Symmetric Nonnegative Matrix Factorization

The Global Optimization Geometry of Shallow Linear Neural Networks

Contact Info

Product

Resources

About