Automatic model redundancy reduction for fast back-propagation for deep neural networks in speech recognition

Abstract

Although deep neural networks (DNNs) have achieved great performance gain, the immense computational cost of DNN model training has become a major block to utilize massive speech data for DNN training. Previous research on DNN training acceleration mostly focussed on hardware-based parallelization. In this paper, node pruning and arc restructur ing are proposed to explore model redundancy after a novel lightly discriminative pretraining process. With some measures of node/arc importance, model redundancies are automatically removed to form a much more compact DNN. This significantly accelerates the subsequent back-propagation (BP) training pro cess. Model redundancy reduction can be combined with multiple GPU parallelization to achieve further acceleration. Experiments showed that the combined acceleration framework can achieve about 85% model size reduction and over 4.2 times speed-up factor for BP training on 2 GPUs, at no loss of recognition accuracy.

Defect Identification Method for Transformer End Pad Falling Based on Acoustic Stability Feature Analysis

“…Existing research on the mechanical characteristics and spectrum of windings is mostly based on steady-state vibration signals. However, vibration sensors in transformers are often difficult to install in substations, and the acoustic signals transmitted through the oil tank from the internal vibrations of transformers are easier to monitor and can contain more comprehensive information than vibration signals [ 18 , 19 ].…”

Binary Deep Neural Networks for Speech Recognition

“…Yu et al [4] incorporated a soft regularization during training to minimize the number of nonzero elements of weight matrices. He et al [5] and Qian et al [6] proposed a method based on node pruning and arc restructuring to prune DNNs for fast inference. Han et al [7] explored an iterative process to prune unimportant connections of DNNs, which reduced the storage and computation by an order of magnitude.…”