Exploiting Efficient Representations in Large-Scale Tensor Decompositions

Abstract

Decomposing tensors into simple terms is often an essential step to discover and understand underlying processes or to compress data. However, storing the tensor and computing its decomposition is challenging in a large-scale setting. Though, in many cases a tensor is structured, i.e., it can be represented using few parameters: a sparse tensor is determined by the positions and values of its nonzeros, a polyadic decomposition by its factor matrices, a Tensor Train by its core tensors, a Hankel tensor by its generating vector, etc. The complexity of tensor decomposition algorithms can be reduced significantly in terms of time and memory if these efficient representations are exploited directly. Only few core operations such as norms and inner products need to be specialized to achieve this, thereby avoiding the explicit construction of multiway arrays. To improve the interpretability of tensor models, constraints are often imposed or multiple datasets are fused through joint factorizations. While imposing these constraints prohibits the use of traditional compression techniques, our framework allows constraints and compression, as well as other efficient representations, to be handled trivially as the underlying optimization variables do not change. To illustrate this, large-scale nonnegative tensor factorization is performed using MLSVD and Tensor Train compression. We also show how vector and matrix data can be analyzed using tensorization while keeping a vector or matrix complexity through the concept of implicit tensorization, as illustrated for Hankelization and Löwnerization. The concepts and numerical properties are extensively investigated with experiments.

Numerical Optimization-Based Algorithms for Data Fusion

“…These mtkrprod can be computed efficiently without explicitly constructing the Khatri-Rao products C B and so on. Various implementations emerged that avoid permuting the tensor in memory [57,58], that avoid communication [59], that reuse intermediate results [58], that exploit sparsity by only computing rows of the Khatri-Rao product corresponding to nonzero entries [60,61], or that exploit structure in the tensor such as tensor train or Hankel structure [60,62].…”

“…Consider, for example, a nonnegativity constraint, i.e., A ≥ 0, in which ≥ holds entry-wise, and the compression matrix U such that U = A, then the constraint ≥ 0 does not imply that A ≥ 0. The structured tensor decomposition framework proposed in [62] avoids this by exploiting the efficient representation of a tensor while keeping the original factor matrices, i.e., A, B and C, in the optimization problem. Instead of working with the original T , its truncated MLSVD S…”

“…Figure 14 illustrates the difference with the candelinc model. Efficient implementations that exploit the multilinear structure of the MLSVD are, e.g., given in [60,62,89]. The structured tensor framework also allows other compression types such as TT compression [62].…”

“…Tensors are often structured and can therefore be represented by few parameters, e.g., all values in a CPD depend on the factor matrices, all entries in a sparse tensor are defined by the positions and values of nonzeros, and a Hankel tensor is determined by one generating vector. By designing specialized algorithms, the per-iteration complexity can be reduced from proportional to the number of entries to proportional to the number of parameters [62].…”

“…In the case the tensor is given by the factors of a decomposition such as a CPD, an MLSVD or a TT, the multilinear structure can be exploited when computing, e.g., the inner product, norm and mtkrprod, without actually constructing the full tensor [60,62,89]. A randomized technique requiring a CPD as input tensor is presented in [101].…”