Relative Error Tensor Low Rank Approximation

Abstract

Large language models have become ubiquitous in modern life, finding applications in various domains such as natural language processing, language translation, and speech recognition. Recently, a breakthrough work [Zhao, Panigrahi, Ge, and Arora Arxiv 2023] explains the attention model from probabilistic context-free grammar (PCFG). One of the central computation task for computing probability in PCFG is formulating a particular tensor low rank approximation problem, we can call it tensor cycle rank. Given an n × n × n third order tensor A, we say that A has cycle rank-k if there exists three n × k 2 size matrices U, V , and W such that for each entry in eachFor the tensor classical rank, tucker rank and train rank, it has been well studied in [Song, Woodruff, Zhong SODA 2019]. In this paper, we generalize the previous "rotation and sketch" technique in page 186 of [Song, Woodruff, Zhong SODA 2019] and show an input sparsity time algorithm for cycle rank.

Randomized Algorithms for Computation of Tucker Decomposition and Higher Order SVD (HOSVD)

“…Other sampling techniques for the HOSVD and the HOOI algorithms can be found in [33], [34], [37]. The proposed algorithms in [33] are based on the sparsification idea where in the first step, a sparse tensor is generated from the original data tensor.…”

“…The idea of the stochastic gradient descent is used in [37], where at each iteration only a subtensor of the original tensor is considered. The randomized algorithms proposed in [34] are based on sampling fibers instead of components.…”

“…Algorithms based on sampling slices are proposed in[34],[92],[93], but those in[92],[93] are not related to the Tucker decomposition.10 VOLUME , 2020…”

Fast and Accurate Randomized Algorithms for Low-rank Tensor Decompositions

“…Aggour et al [1] introduce adaptive sketching for CP decomposition. Song et al [51] discuss the theoretical relative error of various tensor decompositions based on sketching. The work by Cheng et al [13] and Larsen and Kolda [31] accelerate CP-ALS based on leverage score sampling.…”

A near-optimal algorithm for approximating the John Ellipsoid

“…We build our even faster algorithm via sketching techniques. Sketching has been successfully applied to speed up different problems, such as linear programs [LSZ19], clustering [CEM + 15, SYZ18], low rank approximations [CW13, NN13, BW14, CW15a, RSW16, SWZ17], linear regression [CW13, NN13, CLM + 15, CW15b, PSW17, ALS + 18], total least regression [DSWY19], tensor regression [LHW17, DSSW18] and tensor decomposition [WTSA15, SWZ16,SWZ19]. Readers may refer to [Woo14] for a comprehensive survey on sketching technique.…”