Accelerated CNN Training through Gradient Approximation

Harsha, N.; Wang, Ziheng; Amarasinghe, Saman

doi:10.1109/emc249363.2019.00014

Cited by 3 publications

(3 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…of the feature map gradient tensors. (Wang and Nelaturu, 2019). offers a scaling approach to approximate the filter gradient tensors.…”

Section: Comparison To Related Workmentioning

confidence: 99%

A Demand-Side Load Event Detection Algorithm Based on Wide-Deep Neural Networks and Randomized Sparse Backpropagation

Li¹,

Liang²,

Zhao³

et al. 2021

Front. Energy Res.

View full text Add to dashboard Cite

Event detection is an important application in demand-side management. Precise event detection algorithms can improve the accuracy of non-intrusive load monitoring (NILM) and energy disaggregation models. Existing event detection algorithms can be divided into four categories: rule-based, statistics-based, conventional machine learning, and deep learning. The rule-based approach entails hand-crafted feature engineering and carefully calibrated thresholds; the accuracies of statistics-based and conventional machine learning methods are inferior to the deep learning algorithms due to their limited ability to extract complex features. Deep learning models require a long training time and are hard to interpret. This paper proposes a novel algorithm for load event detection in smart homes based on wide and deep learning that combines the convolutional neural network (CNN) and the soft-max regression (SMR). The deep model extracts the power time series patterns and the wide model utilizes the percentile information of the power time series. A randomized sparse backpropagation (RSB) algorithm for weight filters is proposed to improve the robustness of the standard wide-deep model. Compared to the standard wide-deep, pure CNN, and SMR models, the hybrid wide-deep model powered by RSB demonstrates its superiority in terms of accuracy, convergence speed, and robustness.

show abstract

“…of the feature map gradient tensors. (Wang and Nelaturu, 2019). offers a scaling approach to approximate the filter gradient tensors.…”

Section: Comparison To Related Workmentioning

confidence: 99%

A Demand-Side Load Event Detection Algorithm Based on Wide-Deep Neural Networks and Randomized Sparse Backpropagation

Li¹,

Liang²,

Zhao³

et al. 2021

Front. Energy Res.

View full text Add to dashboard Cite

show abstract

“…Given probing vectors drawn from (7), we have to modify the scaling factor of the multi-channel randomized trace estimator (6) to ensure it is unbiased,…”

Section: Single Channel Case Let Us Start By Writing the Action Of A ...mentioning

confidence: 99%

“…Unfortunately, restricting memory usage without introducing significant computational overhead remains a challenge and can lead to difficult to manage additional complexity. Examples include (optimal) checkpointing [1,2], where the state is periodically stored and recomputed during the backward pass, invertible networks [3][4][5], where the state can be derived from the output, and certain approximation methods where computations are made with limited precision arithmetic [6] or where unbiased estimates are made of the gradient using certain approximations [7,8], e.g., via randomized automatic differentiation (RAD, [9]) or via direct feedback alignment (DFA, [10][11][12]).…”

Section: Introductionmentioning

confidence: 99%

Low-memory stochastic backpropagation with multi-channel randomized trace estimation

Louboutin¹,

Siahkoohi²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Thanks to the combination of state-of-the-art accelerators and highly optimized open software frameworks, there has been tremendous progress in the performance of deep neural networks. While these developments have been responsible for many breakthroughs, progress towards solving large-scale problems, such as video encoding and semantic segmentation in 3D, is hampered because access to on-premise memory is often limited. Instead of relying on (optimal) checkpointing or invertibility of the network layers-to recover the activations during backpropagation-we propose to approximate the gradient of convolutional layers in neural networks with a multi-channel randomized trace estimation technique. Compared to other methods, this approach is simple, amenable to analyses, and leads to a greatly reduced memory footprint. Even though the randomized trace estimation introduces stochasticity during training, we argue that this is of little consequence as long as the induced errors are of the same order as errors in the gradient due to the use of stochastic gradient descent. We discuss the performance of networks trained with stochastic backpropagation and how the error can be controlled while maximizing memory usage and minimizing computational overhead.Preprint. Under review.

show abstract

Deep Neural Networks-Based Weight Approximation and Computation Reuse for 2-D Image Classification

et al. 2022

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) are computationally and memory intensive, which present a big challenge for hardware, especially for resource-constrained devices such as Internet-of-Things (IoT) nodes. This paper introduces a new method to improve DNNs performance by fusing approximate computing with data reuse techniques for image recognition applications. First, starting from the pre-trained network, then the DNNs weights are approximated based on the linear and quadratic approximation methods during the retraining phase to reduce the DNN model size and number of arithmetic operations. Then, the DNNs weights are replaced with the linear/quadratic coefficients to execute the inference so that different DNNs weights can be computed using the same coefficients. That leads to a repetition of the weights, which enables the reuse of the DNN sub-computations (computational reuse) and leverages the same data (data reuse) to reduce DNNs computations memory accesses, and improve energy efficiency, albeit at the cost of increased training time. Complete analysis for MNIST, Fashion MNIST, CIFAR 10, CIFAR 100, and tiny ImageNet datasets is presented for image recognition, where different DNN models are used, including LeNet, ResNet, AlexNet, and VGG16. Our results show that the linear approximation achieves 1211.3×, 21.8×, 700×, and 19.3× on LeNet-5 MNIST, LeNet Fashion MNIST, VGG16 and ResNet-20. respectively, with small accuracy loss. Compared to the state-of-the-art Row Stationary (RS) method, the proposed architecture saved 54% of the total number of adders and multipliers needed. Overall, the proposed approach is suitable for IoT edge devices as it reduces computing complexity, memory size, and memory access with a small impact on accuracy.INDEX TERMS DNN, IoT, data reuse, computational reuse, approximate computing.

show abstract

Accelerated CNN Training through Gradient Approximation

Cited by 3 publications

References 4 publications

A Demand-Side Load Event Detection Algorithm Based on Wide-Deep Neural Networks and Randomized Sparse Backpropagation

A Demand-Side Load Event Detection Algorithm Based on Wide-Deep Neural Networks and Randomized Sparse Backpropagation

Low-memory stochastic backpropagation with multi-channel randomized trace estimation

Deep Neural Networks-Based Weight Approximation and Computation Reuse for 2-D Image Classification

Contact Info

Product

Resources

About