A Randomized Algorithm for Principal Component Analysis

Rokhlin, Vladimir; Szlam, Arthur; Tygert, Mark

doi:10.1137/080736417

Cited by 373 publications

(345 citation statements)

References 32 publications

Supporting

Mentioning

335

Contrasting

Order By: Relevance

“…complexity of the coding step is thus similarly reduced when replacing (b) or (c) estimators in (21), but the latter option has a memory usage in O(n k). Although estimators (c) are slightly less performant in the first epochs, they are a good compromise between resource usage and convergence.…”

Section: All Others Estimators {Gmentioning

confidence: 86%

“…Solving (24) is made closer and closer to solving (21), to ensure the correctness of the algorithm (see Section IV). Yet, computing the estimators (b) is no more costly than computing (a) and still permits to speed up a single iteration close to r times.…”

Section: All Others Estimators {Gmentioning

confidence: 99%

“…In this context, models can be learned using only random data summaries, also called sketches. For instance, randomized methods [see 20, for a review] are efficient to compute PCA [21], a classic matrix-factorization approach, and to solve constrained or penalized least-square problems [22,23]. On a theoretical…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Stochastic Subsampling for Factorizing Huge Matrices

Mensch

Mairal

Thirion

et al. 2018

IEEE Trans. Signal Process.

View full text Add to dashboard Cite

Abstract-We present a matrix-factorization algorithm that scales to input matrices with both huge number of rows and columns. Learned factors may be sparse or dense and/or nonnegative, which makes our algorithm suitable for dictionary learning, sparse component analysis, and non-negative matrix factorization. Our algorithm streams matrix columns while subsampling them to iteratively learn the matrix factors. At each iteration, the row dimension of a new sample is reduced by subsampling, resulting in lower time complexity compared to a simple streaming algorithm. Our method comes with convergence guarantees to reach a stationary point of the matrix-factorization problem. We demonstrate its efficiency on massive functional Magnetic Resonance Imaging data (2 TB), and on patches extracted from hyperspectral images (103 GB). For both problems, which involve different penalties on rows and columns, we obtain significant speed-ups compared to state-of-the-art algorithms.

show abstract

Section: All Others Estimators {Gmentioning

confidence: 86%

Section: All Others Estimators {Gmentioning

confidence: 99%

See 1 more Smart Citation

Stochastic Subsampling for Factorizing Huge Matrices

Mensch

Mairal

Thirion

et al. 2018

IEEE Trans. Signal Process.

View full text Add to dashboard Cite

show abstract

“…The input matrices are randomly generated with size ranging from 1000 2 to 22000 2 and rank ranging from 100 to 1000. Figure 3(a) shows a performance comparison of our GPU and CPU implementations to Tygert SVD [10], which is a very fast approximate SVD algorithm that exploits random projection. Here we set the size of the test matrices to range from 1, 000 2 to 22, 000 2 , and the rank to range from 100 to 1000.…”

Section: Resultsmentioning

confidence: 99%

“…For testing and evaluation, we compared results of our GPU-based algorithm to the following three implementations: 1) a highly-optimized, multi-threaded CPU version of QUIC-SVD implemented using Intel Math Kernal Library; 2) MATLAB svds routine; and 3) the Tygert SVD [10], which is a fast CPU-based approximate SVD algorithm built upon random projection. In each test case, we plot the running time as well as the speedup over a range of matrix sizes and ranks.…”

Section: Resultsmentioning

confidence: 99%

A GPU-Based Approximate SVD Algorithm

Foster

Mahadevan

Wang

2012

Parallel Processing and Applied Mathematics

View full text Add to dashboard Cite

Abstract. Approximation of matrices using the Singular Value Decomposition (SVD) plays a central role in many science and engineering applications. However, the computation cost of an exact SVD is prohibitively high for very large matrices. In this paper, we describe a GPU-based approximate SVD algorithm for large matrices. Our method is based on the QUIC-SVD introduced by [6], which exploits a tree-based structure to efficiently discover a subset of rows that spans the matrix space. We describe how to map QUIC-SVD onto the GPU, and improve its speed and stability using a blocked Gram-Schmidt orthogonalization method. Results show that our GPU algorithm achieves 6∼7 times speedup over an optimized CPU version of QUIC-SVD, which itself is orders of magnitude faster than exact SVD methods. Using a simple matrix partitioning scheme, we have extended our algorithm to out-of-core computation, suitable for very large matrices that exceed the main memory size.

show abstract

Large‐scale hydraulic tomography and joint inversion of head and tracer data using the Principal Component Geostatistical Approach (PCGA)

Lee

Kitanidis

2014

Water Resources Research

112

View full text Add to dashboard Cite

The stochastic geostatistical inversion approach is widely used in subsurface inverse problems to estimate unknown parameter fields and corresponding uncertainty from noisy observations. However, the approach requires a large number of forward model runs to determine the Jacobian or sensitivity matrix, thus the computational and storage costs become prohibitive when the number of unknowns, m, and the number of observations, n increase. To overcome this challenge in large-scale geostatistical inversion, the Principal Component Geostatistical Approach (PCGA) has recently been developed as a ''matrixfree'' geostatistical inversion strategy that avoids the direct evaluation of the Jacobian matrix through the principal components (low-rank approximation) of the prior covariance and the drift matrix with a finite difference approximation. As a result, the proposed method requires about K runs of the forward problem in each iteration independently of m and n, where K is the number of principal components and can be much less than m and n for large-scale inverse problems. Furthermore, the PCGA is easily adaptable to different forward simulation models and various data types for which the adjoint-state method may not be implemented suitably. In this paper, we apply the PCGA to representative subsurface inverse problems to illustrate its efficiency and scalability. The low-rank approximation of the large-dimensional dense prior covariance matrix is computed through a randomized eigen decomposition. A hydraulic tomography problem in which the number of observations is typically large is investigated first to validate the accuracy of the PCGA compared with the conventional geostatistical approach. Then the method is applied to a largescale hydraulic tomography with 3 million unknowns and it is shown that underlying subsurface structures are characterized successfully through an inversion that involves an affordable number of forward simulation runs. Lastly, we present a joint inversion of head and tracer test data using MODFLOW and MT3DMS as coupled black-box forward simulation solvers. These applications demonstrate the advantages of the PCGA, i.e., the scalability to high-dimensional inverse problems and the ability to utilize multiple forward models as black boxes.

show abstract

A Randomized Algorithm for Principal Component Analysis

Cited by 373 publications

References 32 publications

Stochastic Subsampling for Factorizing Huge Matrices

Stochastic Subsampling for Factorizing Huge Matrices

A GPU-Based Approximate SVD Algorithm

Large‐scale hydraulic tomography and joint inversion of head and tracer data using the Principal Component Geostatistical Approach (PCGA)

Contact Info

Product

Resources

About