Tensor Contraction Layers for Parsimonious Deep Nets

Kossaifi, Jean; Khanna, Aran; Furlanello, Tommaso; Anandkumar, Anima

doi:10.1109/cvprw.2017.243

Cited by 40 publications

(25 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In fact, HOSVD decomposition is a multidimensional extension of PCA. Some results can be seen in recent papers, such as Hu [64].…”

Section: F Various Types Of Decomposition Applicationsmentioning

confidence: 83%

A Survey on Tensor Techniques and Applications in Machine Learning

Wang

et al. 2019

IEEE Access

View full text Add to dashboard Cite

This survey gives a comprehensive overview of tensor techniques and applications in machine learning. Tensor represents higher order statistics. Nowadays, many applications based on machine learning algorithms require a large amount of structured high-dimensional input data. As the set of data increases, the complexity of these algorithms increases exponentially with the increase of vector size. Some scientists found that using tensors instead of the original input vectors can effectively solve these high-dimensional problems. This survey introduces the basic knowledge of tensor, including tensor operations, tensor decomposition, some tensor-based algorithms, and some applications of tensor in machine learning and deep learning for those who are interested in learning tensors. The tensor decomposition is highlighted because it can effectively extract structural features of data and many algorithms and applications are based on tensor decomposition. The organizational framework of this paper is as follows. In part one, we introduce some tensor basic operations, including tensor decomposition. In part two, applications of tensor in machine learning and deep learning, including regression, supervised classification, data preprocessing, and unsupervised classification based on low rank tensor approximation algorithms are introduced detailly. Finally, we briefly discuss urgent challenges, opportunities and prospects for tensor.

show abstract

“…In fact, HOSVD decomposition is a multidimensional extension of PCA. Some results can be seen in recent papers, such as Hu [64].…”

Section: F Various Types Of Decomposition Applicationsmentioning

confidence: 83%

A Survey on Tensor Techniques and Applications in Machine Learning

Wang

et al. 2019

IEEE Access

View full text Add to dashboard Cite

show abstract

“…A common challenge of the above technique is to determine the tensor rank. Exactly determining a tensor rank in general [49]- [51] Tensor Train [49] CP [55] Tucker [53], [55], [57] Tensor Train [52], [54], [55] CP [48] Tucker Tensor Train is NP-hard [47]. Therefore, in practice one often leverages numerical optimization or statistical techniques to obtain a reasonable rank estimation.…”

Section: Compact Deep Learning Modelsmentioning

confidence: 99%

Tensor Methods for Generating Compact Uncertainty Quantification and Deep Learning Models

Cui

Hawkins

Zhang

2019

2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)

View full text Add to dashboard Cite

Tensor methods have become a promising tool to solve high-dimensional problems in the big data era. By exploiting possible low-rank tensor factorization, many high-dimensional model-based or data-driven problems can be solved to facilitate decision making or machine learning. In this paper, we summarize the recent applications of tensor computation in obtaining compact models for uncertainty quantification and deep learning. In uncertainty analysis where obtaining data samples is expensive, we show how tensor methods can significantly reduce the simulation or measurement cost. To enable the deployment of deep learning on resource-constrained hardware platforms, tensor methods can be used to significantly compress an overparameterized neural network model or directly train a smallsize model from scratch via optimization or statistical techniques. Recent Bayesian tensorized neural networks can automatically determine their tensor ranks in the training process.

show abstract

“…We can also preserve the multi-linear structure in the activation tensor, using tensor contraction (Kossaifi et al, 2017), or by removing fully connected layers and flattening layers altogether and replacing with tensor regression layers (Kossaifi et al, 2018). Adding a stochastic regularization on the rank of the decomposition can also help render the models more robustly (Kolbeinsson et al, 2019).…”

Section: Other Modelsmentioning

confidence: 99%

Spectral Learning on Matrices and Tensors

Janzamin¹,

Ge²,

Kossaifi³

et al. 2019

Self Cite

View full text Add to dashboard Cite

Spectral methods have been the mainstay in several domains such as machine learning, applied mathematics and scientific computing. They involve finding a certain kind of spectral decomposition to obtain basis functions that can capture important structures or directions for the problem at hand. The most common spectral method is the principal component analysis (PCA). It utilizes the principal components or the top eigenvectors of the data covariance matrix to carry out dimensionality reduction as one of its applications. This data pre-processing step is often effective in separating signal from noise. PCA and other spectral techniques applied to matrices have several limitations. By limiting to only pairwise moments, they are effectively making a Gaussian approximation on the underlying data. Hence, they fail on data with hidden variables which lead to non-Gaussianity. However, in almost any data set, there are latent effects that cannot be directly observed, e.g., topics in a document corpus, or underlying causes of a disease. By extending the spectral decomposition

show abstract

Tensor Contraction Layers for Parsimonious Deep Nets

Cited by 40 publications

References 7 publications

A Survey on Tensor Techniques and Applications in Machine Learning

A Survey on Tensor Techniques and Applications in Machine Learning

Tensor Methods for Generating Compact Uncertainty Quantification and Deep Learning Models

Spectral Learning on Matrices and Tensors

Contact Info

Product

Resources

About