LightNorm: Area and Energy-Efficient Batch Normalization Hardware for On-Device DNN Training

Noh, Seock-Hwan; Junsang, Park,; Park, Dahoon; Koo, Jahyun; Choi, Jeik; Kung, Jaeha

doi:10.1109/iccd56317.2022.00072

Cited by 6 publications

(3 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…BN layers are used to resolve issues such as gradient disappearance and gradient explosion in the network during the training process. However, during the forward propagation process, additional intermediate variables

{\bm{\tilde{Z}}}

are generated and the model's efficiency is reduced due to the need to calculate Equations (11)–(13) in sequence [25].…”

Section: Sparse Methods For Deep Learning Modelsmentioning

confidence: 99%

Fault identification for power transformer based on dissolved gas in oil data using sparse convolutional neural networks

Liu,

He,

Liu

et al. 2023

IET Generation Trans & Dist

View full text Add to dashboard Cite

This paper addressed the challenges associated with the complexity, numerous parameters, computational resource demands, and slow processing speed of transformer fault identification models based on deep learning technologies. Sparse convolutional neural network (CNN) approach is proposed for identifying faults related to dissolved gases in oil. Leveraging an improved Gramian angular field, one‐dimensional fault samples are converted into two‐dimensional feature images and data augmentation is implemented to meet the input requirements of deep learning models. Building upon visual geometry group (VGG)19 and residual networks (ResNet)50 networks for fault diagnosis, sparsity techniques are introduced through pruning, the fusion of convolution layers and batch normalization layers, and parameter quantization. Numerical experiments and performance evaluations on dissolved gas in transformer oil fault data demonstrate that the proposed method effectively reduced model complexity, minimized parameter count, conserved computational resources, and improved processing speed while maintaining a considerable level of fault identification accuracy. This made it applicable to edge computing platforms characterized by small form factors and low power consumption in the power industry.

show abstract

{\bm{\tilde{Z}}}

are generated and the model's efficiency is reduced due to the need to calculate Equations (11)–(13) in sequence [25].…”

Section: Sparse Methods For Deep Learning Modelsmentioning

confidence: 99%

Fault identification for power transformer based on dissolved gas in oil data using sparse convolutional neural networks

Liu,

He,

Liu

et al. 2023

IET Generation Trans & Dist

View full text Add to dashboard Cite

show abstract

“…Unlike the previous two works discussed, ref. [ 13 ] proposes an accelerator that focuses on BN. This work has two contributions; the first is a reduced complexity BN layer named LightNorm.…”

Section: Introductionmentioning

confidence: 99%

“…To evaluate these, a reference point is necessary. In addition to pre-existing work from [ 6 , 11 , 12 , 13 , 14 , 15 ] used as a basis, training is performed with the aid of TensorFlow [ 16 ] and a BNN extension, Larq [ 4 ]. Network weights and parameters are then exported for further processing using the methodology elaborated in the subsequent section.…”

Section: Introductionmentioning

confidence: 99%

Pre-Computing Batch Normalisation Parameters for Edge Devices on a Binarized Neural Network

Phipps,

Shang,

Teo

et al. 2023

Sensors

View full text Add to dashboard Cite

Binarized Neural Network (BNN) is a quantized Convolutional Neural Network (CNN), reducing the precision of network parameters for a much smaller model size. In BNNs, the Batch Normalisation (BN) layer is essential. When running BN on edge devices, floating point instructions take up a significant number of cycles to perform. This work leverages the fixed nature of a model during inference, to reduce the full-precision memory footprint by half. This was achieved by pre-computing the BN parameters prior to quantization. The proposed BNN was validated through modeling the network on the MNIST dataset. Compared to the traditional method of computation, the proposed BNN reduced the memory utilization by 63% at 860-bytes without any significant impact on accuracy. By pre-computing portions of the BN layer, the number of cycles required to compute is reduced to two cycles on an edge device.

show abstract

Optimizing Deep Neural Network with Brain Floating Half-Precision Format

Sajid,

Ros,

Kim

2023

2023 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia)

View full text Add to dashboard Cite

LightNorm: Area and Energy-Efficient Batch Normalization Hardware for On-Device DNN Training

Cited by 6 publications

References 7 publications

Fault identification for power transformer based on dissolved gas in oil data using sparse convolutional neural networks

Fault identification for power transformer based on dissolved gas in oil data using sparse convolutional neural networks

Pre-Computing Batch Normalisation Parameters for Edge Devices on a Binarized Neural Network

Optimizing Deep Neural Network with Brain Floating Half-Precision Format

Contact Info

Product

Resources

About