Tensor Methods for Generating Compact Uncertainty Quantification and Deep Learning Models

Abstract

Tensor methods have become a promising tool to solve high-dimensional problems in the big data era. By exploiting possible low-rank tensor factorization, many high-dimensional model-based or data-driven problems can be solved to facilitate decision making or machine learning. In this paper, we summarize the recent applications of tensor computation in obtaining compact models for uncertainty quantification and deep learning. In uncertainty analysis where obtaining data samples is expensive, we show how tensor methods can significantly reduce the simulation or measurement cost. To enable the deployment of deep learning on resource-constrained hardware platforms, tensor methods can be used to significantly compress an overparameterized neural network model or directly train a smallsize model from scratch via optimization or statistical techniques. Recent Bayesian tensorized neural networks can automatically determine their tensor ranks in the training process.

Bayesian tensorized neural networks with automatic rank selection

“…To complete the setup of Bayesian model (15), we still need to specify the prior of rank-control hyper parameters Λ = λ (for CP) or Λ = {λ (n) } d n=1 (for Tucker, TT and TTM). Note that small elements in λ and λ (n) leads to rank reductions in the tensor models, therefore we choose two hyper-prior densities that place high probability near zero.…”

“…Next we discuss how to estimate the resulting posterior density (15). We develop an approach based on stochastic variational inference (SVI) [27] that is compatible with the large-scale stochastic optimization required to train large neural networks.…”

“…[12], [13]. Among these techniques, low-rank tensor compression [12], [13], [14], [15] has achieved possibly the most significant compression, leading to promising reduction of FLOPS and hardware cost [12], [16], [17]. The recent progress of hardware design and algorithm/hardware co-design [16], [17], [18], [19], [20] of tensor operations can further reduce the run-time and boost the energy efficiency of tensorized models on clusters, FPGA and ASIC chips.…”

SiamEvent: Event-based Object Tracking via Edge-aware Similarity Learning with Siamese Networks

“…For k, we choose the trilinar kernel [47] for SiamEvent as it shows the best performance based on the experiments. As processing 4D tensor, e.g., 3D convolution, needs high computational cost and may degrade the test speed [48], we then reshape S ± to a 3D tensor S by stacking the voxel grids of two polarities. In such a way, the tensor can be fed to our proposed framework to track an arbitrary target.…”