An Efficient Gauss--Newton Algorithm for Symmetric Low-Rank Product Matrix Approximations

Abstract

We derive and study a Gauss-Newton method for computing a symmetric lowrank product XX T , where X ∈ R n×k for k < n, that is the closest to a given symmetric matrix A ∈ R n×n in Frobenius norm. When A = B T B (or BB T ), this problem essentially reduces to finding a truncated singular value decomposition of B. Our Gauss-Newton method, which has a particularly simple form, shares the same order of iteration-complexity as a gradient method when k n, but can be significantly faster on a wide range of problems. In this paper, we prove global convergence and a Q-linear convergence rate for this algorithm and perform numerical experiments on various test problems, including those from recently active areas of matrix completion and robust principal component analysis. Numerical results show that the proposed algorithm is capable of providing considerable speed advantages over Krylov subspace methods on suitable application problems where high-accuracy solutions are not required. Moreover, the algorithm possesses a higher degree of concurrency than Krylov subspace methods, thus offering better scalability on modern multi-/manycore computers. Introduction.Low-rank matrix approximation techniques, such as dominant eigenvalue or singular value decompositions (SVDs), appear in a wide variety of scientific and engineering applications. For example, principal component analyses (PCA) in statistics use them to compute a few directions of maximal variance. Many data dimensionality reduction methods in machine learning use them to perform low-dimensional embedding of high-dimensional data. More recently, identifying dominant eigenvalue or SVDs of a sequence of closely related matrices has become an indispensable algorithmic component for many first-order optimization methods for various convex optimization problems, such as semidefinite programming, low-rank matrix completion, robust principal component analysis, sparse principal component analysis, sparse inverse covariance matrix estimation, and nearest correlation matrix estimation (see [9] for a more detailed summary). More often than not, the computational cost of doing low-rank approximations forms a major bottleneck in the overall efficiency of solution processes.

A Brief Introduction to Manifold Optimization

“…In particular, the efficiency may not be significantly improved when working with thousands of CPU cores. From the perspective of optimization, a series of fast algorithms for solving (2.9) was proposed in [65,64,93,96], whose essential parts can be divided into two steps, updating a subspace to approximate the eigenvector space better and extracting eigenvectors by the Rayleigh-Ritz (RR) process. The main numerical algebraic technique for updating subspaces is usually based on the Krylov subspace, which constructs a series of orthogonal bases sequentially.…”

“…More importantly, the model allows one to design an algorithm that uses only matrix-matrix multiplication. A Gauss-Newton algorithm for calculating low rank decomposition is developed in [65]. When the matrix to be decomposed is of low rank, this algorithm can be more effective while its complexity is similar to the gradient method but with Q linear convergence.…”

Algorithms for Stable PCA

“…Oiter (rmn), Opre=unknown, Oconv(1/µT ) (IALM 13 ) Lin et al [91] Partial SVD, Linear Time SVD [198] Limited Memory SVD (LMSVD 14 ) [98] Symmetric Low-Rank Product-Gauss-Newton [99] Alternating Direction Method (ADM)…”

On the Applications of Robust PCA in Image and Video Processing

“…First, new Singular Value Decomposition (SVD) solutions have been developed to make the iterations as efficient as possible and to deal with the fact that the standard SVD solution fails if the data are corrupted by anything other than small noise. For example, approximated SVD solutions exist to avoid full SVD computations in order to reduce computation time such as partial SVD algorithms [170], linear time SVD algorithms [303], limited memory SVD algorithms [178], symmetric low-rank product-Gauss-Newton algorithms [179], Block Lanczos with Warm Start (BLWS) algorithms [172], and randomized SVD algorithms [83] [335] [145]. Moreover, a lot of video data arrive sequentially over time and the subspace in which the data lie can change with time.…”