Residual Matrix Product State for Machine Learning

Meng, Ye-Ming; Zhang, Jing; Zhang, Peng; Gao, Chao; Ran, Shi-Ju

doi:10.48550/arxiv.2012.11841

Cited by 4 publications

(7 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…which have been used in other implementations of tensor network regression [4,15,16]. When equation ( 7) is used to construct X, the regression function ⃗ f (⃗ x; W) from equation (6) becomes…”

Section: Regression With Tensorsmentioning

confidence: 99%

“…This could be viewed as a form of data tensor preprocessing, where elements of X corresponding to interaction degrees other than j are removed. Alternatively, the tensor diagrams in equation (16) show that it is equally valid to consider P ( j) acting on the weight tensor W. Under this interpretation, the set { ⃗ d ( j) (⃗ x; W)} m j=0 consists of regressions on X performed by different models, each derived from a common tensor W by keeping only those elements corresponding to interactions of degree j. We will shift between these two interpretations freely throughout the remainder of this section, describing the interaction decomposition as a procedure which picks out different pieces of a tensor network model and which restricts the set of feature products that can be used for regression.…”

Section: Interaction Subspacesmentioning

confidence: 99%

See 1 more Smart Citation

Interaction decompositions for tensor network regression

Convy

Whaley

2022

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

It is well known that tensor network regression models operate on an exponentially large feature space, but questions remain as to how effectively they are able to utilize this space. Using a polynomial featurization, we propose the interaction decomposition as a tool that can assess the relative importance of different regressors as a function of their polynomial degree. We apply this decomposition to tensor ring and tree tensor network models trained on the MNIST and Fashion MNIST datasets, and find that up to 75% of interaction degrees are contributing meaningfully to these models. We also introduce a new type of tensor network model that is explicitly trained on only a small subset of interaction degrees, and find that these models are able to match or even outperform the full models using only a fraction of the exponential feature space. This suggests that standard tensor network models utilize their polynomial regressors in an inefficient manner, with the lower degree terms being vastly under-utilized.

show abstract

Section: Regression With Tensorsmentioning

confidence: 99%

Section: Interaction Subspacesmentioning

confidence: 99%

Interaction decompositions for tensor network regression

Convy

Whaley

2022

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

show abstract

“…We will now discuss the application of deep tensor networks on the MNIST and Fashion MNIST image classification tasks. Tensor networks have previously been applied to both tasks with moderate success [1,2,3,4,5,6,7,8,36]. The best results using tensor network methods in combination with neural networks are 0.69% error rate on MNIST [36] (CNN+PEPS) and 8.9% error rate on Fashion MNIST [36] (CNN+PEPS).…”

Section: Image Classification With Deep Tensor Networkmentioning

confidence: 99%

“…Motivated by the successful approximation of quantum amplitudes, tensor networks have also been applied to machine learning problems (e.g. image classification [1,2,3,4,5,6,7,8], generative modelling [9,10,11,12], sequence and language modelling [13,14,15,16], anomaly detection [17,18]). Adopting tensor network methods in machine learning enables compression of neural networks [19,20,21], adaptive training algorithms [22], derivation of interesting generalisation bounds [15], information theoretical insight [23,24,25,26], and new connections between machine learning and physics [27,28,29].…”

Section: Introductionmentioning

confidence: 99%

Deep tensor networks with matrix product operators

Žunkovič

2022

Preprint

View full text Add to dashboard Cite

We introduce deep tensor networks, which are exponentially wide neural networks based on the tensor network representation of the weight matrices. We evaluate the proposed method on the image classification (MNIST, FashionMNIST) and sequence prediction (cellular automata) tasks. In the image classification case, deep tensor networks improve our matrix product state baselines and achieve 0.49% error rate on MNIST and 8.3% error rate on FashionMNIST. In the sequence prediction case, we demonstrate an exponential improvement in the number of parameters compared to the onelayer tensor network methods. In both cases, we discuss the non-uniform and the uniform tensor network models and show that the latter generalizes well to different input sizes.

show abstract

“…Thanks to the rapid progress of computer technology, data in tensor format (i.e., multi-dimensional array) are emerging in computer vision, machine learning, remote sensing, quantum physics, and many other fields, triggering an increasing need for tensor-based learning theory and algorithms [1][2][3][4][5][6]. In this paper, we carry out both theoretic and algorithmic research studies on tensor recovery from linear observations, which is a typical problem in tensor learning aiming to learn an unknown tensor when only a limited number of its noisy linear observations are available [7].…”

Section: Introductionmentioning

confidence: 99%

A Hybrid Norm for Guaranteed Tensor Recovery

et al. 2022

View full text Add to dashboard Cite

Benefiting from the superiority of tensor Singular Value Decomposition (t-SVD) in excavating low-rankness in the spectral domain over other tensor decompositions (like Tucker decomposition), t-SVD-based tensor learning has shown promising performance and become an emerging research topic in computer vision and machine learning very recently. However, focusing on modeling spectral low-rankness, the t-SVD-based models may be insufficient to exploit low-rankness in the original domain, leading to limited performance while learning from tensor data (like videos) that are low-rank in both original and spectral domains. To this point, we define a hybrid tensor norm dubbed the “Tubal + Tucker” Nuclear Norm (T2NN) as the sum of two tensor norms, respectively, induced by t-SVD and Tucker decomposition to simultaneously impose low-rankness in both spectral and original domains. We further utilize the new norm for tensor recovery from linear observations by formulating a penalized least squares estimator. The statistical performance of the proposed estimator is then analyzed by establishing upper bounds on the estimation error in both deterministic and non-asymptotic manners. We also develop an efficient algorithm within the framework of Alternating Direction Method of Multipliers (ADMM). Experimental results on both synthetic and real datasets show the effectiveness of the proposed model.

show abstract

Residual Matrix Product State for Machine Learning

Cited by 4 publications

References 20 publications

Interaction decompositions for tensor network regression

Interaction decompositions for tensor network regression

Deep tensor networks with matrix product operators

A Hybrid Norm for Guaranteed Tensor Recovery

Contact Info

Product

Resources

About