Efficient parallel CP decomposition with pairwise perturbation and multi-sweep dimension tree

Ma, Linjian; Solomonik, Edgar

doi:10.1109/ipdps49936.2021.00049

Cited by 6 publications

(2 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The cost of each sweep of Algorithm 3.1 corresponds to the cost of computing the pseudoinverse of each factor, as well as a set of N MTTKRP operations. A dimension tree or multi-sweep dimension tree [37] may be used to compute the set of MTTKRPs in the same way as done in the alternating least squares algorithm. The overall per-sweep cost with a multi-sweep dimension tree is then given by…”

Section: Cost Analysismentioning

confidence: 99%

Alternating Mahalanobis Distance Minimization for Stable and Accurate CP Decomposition

Singh¹,

Solomonik²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

CP decomposition (CPD) is prevalent in chemometrics, signal processing, data mining and many more fields. While many algorithms have been proposed to compute the CPD, alternating least squares (ALS) remains one of the most widely used algorithm for computing the decomposition. Recent works have introduced the notion of eigenvalues and singular values of a tensor and explored applications of eigenvectors and singular vectors in areas like signal processing, data analytics and in various other fields. We introduce a new formulation for deriving singular values and vectors of a tensor by considering the critical points of a function different from what is used in the previous work. Computing these critical points in an alternating manner motivates an alternating optimization algorithm which corresponds to alternating least squares algorithm in the matrix case. However, for tensors with order greater than equal to 3, it minimizes an objective function which is different from the commonly used least squares loss. Alternating optimization of this new objective leads to simple updates to the factor matrices with the same asymptotic computational cost as ALS. We show that a subsweep of this algorithm can achieve a superlinear convergence rate for exact CPD with known rank and verify it experimentally. We then view the algorithm as optimizing a Mahalanobis distance with respect to each factor with ground metric dependent on the other factors. This perspective allows us to generalize our approach to interpolate between updates corresponding to the ALS and the new algorithm to manage the tradeoff between stability and fitness of the decomposition. Our experimental results show that for approximating synthetic and real-world tensors, this algorithm and its variants converge to a better conditioned decomposition with comparable and sometimes better fitness as compared to the ALS algorithm.

show abstract

Section: Cost Analysismentioning

confidence: 99%

Alternating Mahalanobis Distance Minimization for Stable and Accurate CP Decomposition

Singh¹,

Solomonik²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This setting is useful for application of sketching to alternating optimization in tensor-related problems, such as tensor decompositions. In alternating optimization, multiple contraction trees of the data x are chosen in an alternating order to form multiple optimization subproblems, each updating part of the variables [36,24,27]. Designing embeddings under the constraint can help reuse contracted intermediates across subproblems.…”

Section: Introductionmentioning

confidence: 99%

Cost-efficient Gaussian Tensor Network Embeddings for Tensor-structured Inputs

Ma¹,

Solomonik²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

This work discusses tensor network embeddings, which are random matrices (S) with tensor network structure. These embeddings have been used to perform dimensionality reduction of tensor network structured inputs x and accelerate applications such as tensor decomposition and kernel regression. Existing works have designed embeddings for inputs x with specific structures, such as the Kronecker product or Khatri-Rao product, such that the computational cost for calculating Sx is efficient. We provide a systematic way to design tensor network embeddings consisting of Gaussian random tensors, such that for inputs with more general tensor network structures, both the sketch size (row size of S) and the sketching computational cost are low. We analyze general tensor network embeddings that can be reduced to a sequence of sketching matrices. We provide a sufficient condition to quantify the accuracy of such embeddings and derive sketching asymptotic cost lower bounds using embeddings that satisfy this condition and have a sketch size lower than any input dimension. We then provide an algorithm to efficiently sketch input data using such embeddings. The sketch size of the embedding used in the algorithm has a linear dependence on the number of sketching dimensions of the input. Assuming tensor contractions are performed with classical dense matrix multiplication algorithms, this algorithm achieves asymptotic cost within a factor of O( √ m) of our cost lower bound, where m is the sketch size. Further, when each tensor in the input has a dimension that needs to be sketched, this algorithm yields the optimal sketching asymptotic cost. We apply our sketching analysis to inexact tensor decomposition optimization algorithms. We provide a sketching algorithm for CP decomposition that is asymptotically faster than existing work in multiple regimes, and show optimality of an existing algorithm for tensor train rounding.

show abstract

Accelerating alternating least squares for tensor decomposition by pairwise perturbation

Solomonik

2022

Numerical Linear Algebra App

Self Cite

View full text Add to dashboard Cite

The alternating least squares (ALS) algorithm for CP and Tucker decomposition is dominated in cost by the tensor contractions necessary to set up the quadratic optimization subproblems. We introduce a novel family of algorithms that uses perturbative corrections to the subproblems rather than recomputing the tensor contractions. This approximation is accurate when the factor matrices are changing little across iterations, which occurs when ALS approaches convergence. We provide a theoretical analysis to bound the approximation error.Our numerical experiments demonstrate that the proposed pairwise perturbation algorithms are easy to control and converge to minima that are as good as ALS. The experimental results show improvements of up to 3.1× with respect to state-of-the-art ALS approaches for various model tensor problems and real datasets.

show abstract

Efficient parallel CP decomposition with pairwise perturbation and multi-sweep dimension tree

Cited by 6 publications

References 40 publications

Alternating Mahalanobis Distance Minimization for Stable and Accurate CP Decomposition

Alternating Mahalanobis Distance Minimization for Stable and Accurate CP Decomposition

Cost-efficient Gaussian Tensor Network Embeddings for Tensor-structured Inputs

Accelerating alternating least squares for tensor decomposition by pairwise perturbation

Contact Info

Product

Resources

About