Reshaping deep neural network for fast decoding by node-pruning

He, Tianxing; Fan, Yuchen; Ye, Qian; Tan, Tian; Yu, Kai

doi:10.1109/icassp.2014.6853595

Cited by 99 publications

(84 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Automatic node and arc level restructuring algorithms are given. Different from other model restructuring works [8], [9], [10], which performed the model rebuilding at the end of neural network training, we find ways to restructure the DNN before BP to achieve significant accelerations in training speed.…”

Section: Ill Aut Om Atic Model Redundancy Reductionmentioning

confidence: 99%

“…Different from some earlier works of node restructuring [l3], [14], which were applied to small scale shallow neural networks with the motivation of alleviating the over-fitting problem, our previous work [10] investigated how node re structuring can help reduce the complexity of a large scale DNN. Several manual node pruning approaches were per formed on well trained DNNs to accelerate the decoding.…”

Section: B Node Restructuringmentioning

confidence: 99%

“…Moreover, the work in [9] applies singular value decomposition (SVD) on the weight matrices of a fully trained DNN to reduce the model size. Our previous work in [10] investigates node pruning on top of a well trained DNN model. All of these approaches can reduce DNN complexity while keeping the original recognition performance after model retuning.…”

Section: Introductionmentioning

confidence: 99%

“…It has been observed that the speech recognition performance typically improves as the DNN depth and width increase. However recent works show that huge sparseness exists in the DNN model [8], [9], [10]. It is shown in [8] that a large portion of the weight parameters in the DNN are very small, and directly throwing away the weights close to zero can effectively reduce the model size.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Automatic model redundancy reduction for fast back-propagation for deep neural networks in speech recognition

Deng

et al. 2015

2015 International Joint Conference on Neural Networks (IJCNN)

Self Cite

View full text Add to dashboard Cite

Although deep neural networks (DNNs) have achieved great performance gain, the immense computational cost of DNN model training has become a major block to utilize massive speech data for DNN training. Previous research on DNN training acceleration mostly focussed on hardware-based parallelization. In this paper, node pruning and arc restructur ing are proposed to explore model redundancy after a novel lightly discriminative pretraining process. With some measures of node/arc importance, model redundancies are automatically removed to form a much more compact DNN. This significantly accelerates the subsequent back-propagation (BP) training pro cess. Model redundancy reduction can be combined with multiple GPU parallelization to achieve further acceleration. Experiments showed that the combined acceleration framework can achieve about 85% model size reduction and over 4.2 times speed-up factor for BP training on 2 GPUs, at no loss of recognition accuracy.

show abstract

Section: Ill Aut Om Atic Model Redundancy Reductionmentioning

confidence: 99%

Section: B Node Restructuringmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Automatic model redundancy reduction for fast back-propagation for deep neural networks in speech recognition

Deng

et al. 2015

2015 International Joint Conference on Neural Networks (IJCNN)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Optimizations like those found in [34] could also be applied to a network obtained through this approach for further speed-up. Note that unlike weight matrix decomposition and node pruning [40] approaches, soft target training is considerably more flexible because the input feature type and NN architecture are not restricted to be the same as the teacher. Furthermore, an ensemble of networks could be used to relabel inputs for the small network.…”

Section: Training a Small Dnn With Soft Targets From A Large Dnnmentioning

confidence: 99%

Wise teachers train better DNN acoustic models

Price

Iso²,

Sairyo

2016

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

Automatic speech recognition is becoming more ubiquitous as recognition performance improves, capable devices increase in number, and areas of new application open up. Neural network acoustic models that can utilize speaker-adaptive features, have deep and wide layers, or more computationally expensive architectures, for example, often obtain best recognition accuracy but may not be suitable for the given budget of computational and storage resources or latency required by the deployed system. We explore a straightforward training approach which takes advantage of highly accurate but expensive-to-evaluate neural network acoustic models by using their outputs to relabel training examples for easier-to-deploy models. Experiments on a large vocabulary continuous speech recognition task offer relative reductions in word error rate of up to 16.7 % over training with the hard aligned labels by effectively making use of large amounts of additional untranscribed data. Somewhat remarkably, the approach works well even when only two output classes are present. Experiments on a voice activity detection task give relative reductions in equal error rate of up to 11.5 % when using a convolutional neural network to relabel training examples for a feedforward neural network. An investigation into the hidden layer weight matrices finds that soft target-trained networks tend to produce weight matrices having fuller rank and slower decay in singular values than their hard target-trained counterparts, suggesting that more of the network's capacity is utilized for learning additional information giving better accuracy.

show abstract

Neural Networks with Block Diagonal Inner Product Layers

Nesky

Stout

2018

Artificial Neural Networks and Machine Learning – ICANN 2018

View full text Add to dashboard Cite

Reshaping deep neural network for fast decoding by node-pruning

Cited by 99 publications

References 10 publications

Automatic model redundancy reduction for fast back-propagation for deep neural networks in speech recognition

Automatic model redundancy reduction for fast back-propagation for deep neural networks in speech recognition

Wise teachers train better DNN acoustic models

Neural Networks with Block Diagonal Inner Product Layers

Contact Info

Product

Resources

About