Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination

Hawkins, Cole; Liu, Xing; Zhang, Zheng

doi:10.48550/arxiv.2010.08689

Cited by 5 publications

(13 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We follow the notation and exposition of (Hawkins et al, 2020) to describe the low-rank tensor formats that we employ. This paper uses lower-case letters (e.g., a) to denote scalars, bold lowercase letters (e.g., a) to represent vectors, bold uppercase letters (e.g., A) to represent matrices, and bold calligraphic letters (e.g., A) to denote tensors.…”

Section: Tensor Formatsmentioning

confidence: 99%

Low-Rank+Sparse Tensor Compression for Neural Networks

Hawkins¹,

Yang²,

Li³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Low-rank tensor compression has been proposed as a promising approach to reduce the memory and compute requirements of neural networks for their deployment on edge devices. Tensor compression reduces the number of parameters required to represent a neural network weight by assuming network weights possess a coarse higher-order structure. This coarse structure assumption has been applied to compress large neural networks such as VGG and ResNet. However modern state-of-the-art neural networks for computer vision tasks (i.e. MobileNet, Effi-cientNet) already assume a coarse factorized structure through depthwise separable convolutions, making pure tensor decomposition a less attractive approach. We propose to combine low-rank tensor decomposition with sparse pruning in order to take advantage of both coarse and fine structure for compression. We compress weights in SOTA architectures (MobileNetv3, EfficientNet, Vision Transformer) and compare this approach to sparse pruning and tensor decomposition alone.

show abstract

Section: Tensor Formatsmentioning

confidence: 99%

Low-Rank+Sparse Tensor Compression for Neural Networks

Hawkins¹,

Yang²,

Li³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Rank Reduction. Our objective function is the negative log-posterior of the Bayesian model in Hawkins et al (2020). Empirical results shows this prior leads to good trade-off between model size and accuracy, and it is hyper-parameter-free which eliminates the need of hyper-parameter finetuning.…”

Section: Rank-adaptive Tenosirzed Trainingmentioning

confidence: 99%

On-FPGA Training with Ultra Memory Reduction: A Low-Precision Tensor Method

Zhang¹,

Hawkins²,

Zhang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Various hardware accelerators have been developed for energy-efficient and realtime inference of neural networks on edge devices. However, most training is done on high-performance GPUs or servers, and the huge memory and computing costs prevent training neural networks on edge devices. This paper proposes a novel tensor-based training framework, which offers orders-of-magnitude memory reduction in the training process. We propose a novel rank-adaptive tensorized neural network model, and design a hardware-friendly low-precision algorithm to train this model. We present an FPGA accelerator to demonstrate the benefits of this training method on edge devices. Our preliminary FPGA implementation achieves 59× speedup and 123× energy reduction compared to embedded CPU, and 292× memory reduction over a standard full-size training.

show abstract

“…• Secondly, hardware-friendly tensor algorithms should be developed to facilitate their implementation on edge devices. In this section, we summarize our recent work on the algorithm [15,16] and hardware [40] levels to address these challenges.…”

Section: Ultra-low Memory Trainingmentioning

confidence: 99%

“…This can be done with various low-rank tensor decomposition formats as shown in Fig. 1 [15]. In various tensor decompositions, 𝚽 denotes the associated tensor factors.…”

Section: Bayesian Tensorized Training Modelsmentioning

confidence: 99%

“…In various tensor decompositions, 𝚽 denotes the associated tensor factors. For large fully connected layers and embedding tables, the tensor-train matrix (TTM) format turns to be highly effective [15]. In the TTM format, 𝚽 = {G (𝑛) } 𝑑 𝑛=1 , and each G (𝑛) ∈ R 𝑅 𝑛−1 ×𝐽 𝑛 ×𝐼 𝑛 ×𝑅 𝑛 is an order-4 TTM core.…”

Section: Bayesian Tensorized Training Modelsmentioning

confidence: 99%

See 1 more Smart Citation

3U-EdgeAI

Chen

Hawkins

Zhang

et al. 2021

Proceedings of the 2021 Great Lakes Symposium on VLSI

Self Cite

View full text Add to dashboard Cite

The deep neural network (DNN) based AI applications on the edge require both low-cost computing platforms and high-quality services. However, the limited memory, computing resources, and power budget of the edge devices constrain the effectiveness of the DNN algorithms. Developing edge-oriented AI algorithms and implementations (e.g., accelerators) is challenging. In this paper, we summarize our recent efforts for efficient on-device AI development from three aspects, including both training and inference. First, we present on-device training with ultra-low memory usage. We propose a novel rank-adaptive tensor-based tensorized neural network model, which offers orders-of-magnitude memory reduction during training. Second, we introduce an ultra-low bitwidth quantization method for DNN model compression, achieving the state-of-the-art accuracy under the same compression ratio. Third, we introduce an ultra-low latency DNN accelerator design, practicing the software/hardware co-design methodology. This paper emphasizes the importance and efficacy of training, quantization and accelerator design, and calls for more research breakthroughs in the area for AI on the edge. CCS CONCEPTS• Hardware → Reconfigurable logic and FPGAs; • Computing methodologies → Machine learning.

show abstract

Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination

Cited by 5 publications

References 0 publications

Low-Rank+Sparse Tensor Compression for Neural Networks

Low-Rank+Sparse Tensor Compression for Neural Networks

On-FPGA Training with Ultra Memory Reduction: A Low-Precision Tensor Method

3U-EdgeAI

Contact Info

Product

Resources

About