Tensor Decompositions in Deep Learning

Bacciu, Davide; Mandic, Danilo P.

doi:10.48550/arxiv.2002.11835

Cited by 2 publications

(4 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The popularity of image decomposition and natural language processing in the context of DL [28] has increased the demand for effective storage and support of the numerous parameters needed by deep learning algorithms. One solution to this problem is to employ sparse representation methods to compress the parameter matrix and lower storage pressure, such as tensor decomposition and matrix decomposition.…”

Section: Resultsmentioning

confidence: 99%

“…Vectors can benefit from the compression capabilities of matrix-TDs' reshaping and unfolding techniques, and an analysis is done to find the ideal compress ratio by reshaping. In DL [28], the input vector is reshaped into a three-way tensor, and the output tensor is unfolded into a vector. This reshaping and unfolding process reduces the number of parameters through tensor decomposition.…”

Section: Resultsmentioning

confidence: 99%

“…The input vectors can be reshaped into an N-way tensor, while the output tensor of the compressed neural network is also N-way. The goal is to minimize the compressed network's number of parameters, which can be formulated as an optimization problem [28].…”

Section: Resultsmentioning

confidence: 99%

See 2 more Smart Citations

<strong></strong>How Satellite Imaging and Deep Learning Are Influenced by Tensor Decompositions: A Review

A. Mageed

2024

Preprint

View full text Add to dashboard Cite

This study focuses on examining various approaches for Tensor Decompositions (TDs) and their potential applications in satellite imaging (SI) and deep learning (DL). The research highlights how these decompositions can contribute to the advancement of SI and DL, providing valuable insights for future utilization of TDs in research. It explores how these techniques contribute to the advancement of SI and DL methods, providing insights into their potential applications and suggesting future research directions. The study aims to enhance the use of TDs techniques to further advance research efforts in these fields. More importantly, the current investigation offers a thorough analysis of the potential advantages and reasons for employing TDs techniques in different domains, including satellite imaging and deep learning. The aim is to enhance research outcomes by utilizing TDs. The review also identifies unresolved issues and proposes future directions for further investigation in this field. Fundamentally, these proposed open problems will open new grounds to the research community to articulate, innovate, and provide more real-life applications to improve the current state of the art by delving into a wider vision for a higher-level performance of both satellite imaging (SI) and deep learning (DL). Looking at the bigger scenario, this also suggests that TDs could be potentially employed to revolutionize existing machine learning technologies as well as the current space AI industry.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

<strong></strong>How Satellite Imaging and Deep Learning Are Influenced by Tensor Decompositions: A Review

A. Mageed

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Moreover, Sidiropoulos et al investigated tensor factorization models and set up their relationship with signal processing and machine learning applications [12]. Bacciu et al summarized tensor decompositions in deep learning, including TTD, Canonical Polyadic (CP) decomposition, and TD, and also compared their performance on neural model compression [31]. Furthermore, our previous empirical study [19] demonstrated that TD enables a compressed DNN model, but the model cannot guarantee the DNN performance.…”

Section: Related Work and Our Contributionmentioning

confidence: 99%

Exploiting Low-Rank Tensor-Train Deep Neural Networks Based on Riemannian Gradient Descent With Illustrations of Speech Processing

Qi¹,

Yang²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

This work focuses on designing low complexity hybrid tensor networks by considering trade-offs between the model complexity and practical performance. Firstly, we exploit a low-rank tensor-train deep neural network (TT-DNN) to build an end-to-end deep learning pipeline, namely LR-TT-DNN. Secondly, a hybrid model combining LR-TT-DNN with a convolutional neural network (CNN), which is denoted as CNN+(LR-TT-DNN), is set up to boost the performance. Instead of randomly assigning large TT-ranks for TT-DNN, we leverage Riemannian gradient descent to determine a TT-DNN associated with small TT-ranks. Furthermore, CNN+(LR-TT-DNN) consists of convolutional layers at the bottom for feature extraction and several TT layers at the top to solve regression and classification problems. We separately assess the LR-TT-DNN and CNN+(LR-TT-DNN) models on speech enhancement and spoken command recognition tasks. Our empirical evidence demonstrates that the LR-TT-DNN and CNN+(LR-TT-DNN) models with fewer model parameters can outperform the TT-DNN and CNN+(TT-DNN) counterparts.

show abstract

Tensor Decompositions in Deep Learning

Cited by 2 publications

References 17 publications

<strong></strong>How Satellite Imaging and Deep Learning Are Influenced by Tensor Decompositions: A Review

<strong></strong>How Satellite Imaging and Deep Learning Are Influenced by Tensor Decompositions: A Review

Exploiting Low-Rank Tensor-Train Deep Neural Networks Based on Riemannian Gradient Descent With Illustrations of Speech Processing

Contact Info

Product

Resources

About