Efficient Knowledge Distillation from an Ensemble of Teachers

Fukuda, Takashi; Suzuki, Masato; Kurata, Gakuto; Thomas, Samuel; Cui, Jia; Ramabhadran, Bhuvana

doi:10.21437/interspeech.2017-614

Cited by 212 publications

(127 citation statements)

References 11 publications

Supporting

Mentioning

126

Contrasting

Order By: Relevance

“…The VGG model has the same context dependent phones in the output layer as the parent CNN model. Then, the student CNN model was trained as the parent CNN model with a teacher-student training framework with the same 3600 hours of audio data after soft labels were generated from the VGG model [19].…”

Section: Data and Model Buildsmentioning

confidence: 99%

Direct Neuron-Wise Fusion of Cognate Neural Networks

2019

Self Cite

View full text Add to dashboard Cite

This paper proposes a method to create a robust acoustic model by directly fusing multiple neural networks that have dissimilar characteristics without any additional layers/nodes involving retraining procedures. The fused neural networks derive from a shared parent neural network and are referred to as cognate (child) neural networks in this paper. The neural networks are fused by interpolating weight and bias parameters associated with each neuron with a different fusion weight, assuming that cognate neural networks to be fused have the same topology. Therefore, no extra computational cost during decoding is required. The fusion weight is determined by considering a cosine similarity estimated from parameters connecting to the neuron and the fusion is performed for every neuron. Experiments were carried out using a test suite consisting of various acoustic conditions with a wide SNR range, speakers including foreign accented speakers, and speaking styles. From the experiments, the network created by fusing cognate neural networks showed consistent improvement on average compared with the commercialgrade domain-free network originating from the parent model. In addition, we demonstrate that the fusion considering input connections to the neuron achieves the highest accuracy in our experiments.

show abstract

Section: Data and Model Buildsmentioning

confidence: 99%

Direct Neuron-Wise Fusion of Cognate Neural Networks

2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…It performed well in terms of speaker individuality and root mean square log-spectral distortion (RMS-LSD). Additionally, non-learning BWE approaches are also reported [9], [17]- [20] in recent years.…”

Section: Introductionmentioning

confidence: 99%

Blind Bandwidth Extension with a Non-Linear Function and Its Evaluation on Automatic Speaker Verification

Kaminishi

Miyamoto

Shiota

et al. 2020

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

This study evaluates the effects of some non-learning blind bandwidth extension (BWE) methods on state-of-the-art automatic speaker verification (ASV) systems. Recently, a non-linear bandwidth extension (N-BWE) method has been proposed as a blind, non-learning, and lightweight BWE approach. Other non-learning BWEs have also been developed in recent years. For ASV evaluations, most data available to train ASV systems is narrowband (NB) telephone speech. Meanwhile, wideband (WB) data have been used to train the state-of-the-art ASV systems, such as i-vector, d-vector, and x-vector. This can cause sampling rate mismatches when all datasets are used. In this paper, we investigate the influence of sampling rate mismatches in the x-vector-based ASV systems and how non-learning BWE methods perform against them. The results showed that the N-BWE method improved the equal error rate (EER) on ASV systems based on the x-vector when the mismatches were present. We researched the relationship between objective measurements and EERs. Consequently, the N-BWE method produced the lowest EERs on both ASV systems and obtained the lower RMS-LSD value and the higher STOI score.

show abstract

“…Instead of augmenting the data by applying various kinds of signal distortions to the input acoustic features as is often done, our idea is to augment the training data by creating multiple copies with different labels that reflect the corresponding output targets of each task. A similar idea has been recently used in knowledge distillation [25]. They applied the AS to the multi-teacher student approach where update is performed sequencially in each computation of each teacher loss instead of using linear interpolated losses; achieving better performance than models trained from scratch or using interpolated loss functions.…”

Section: Introductionmentioning

confidence: 99%

Multi-task Learning with Augmentation Strategy for Acoustic-to-word Attention-based Encoder-decoder Speech Recognition

et al. 2018

View full text Add to dashboard Cite

In this paper, we propose a novel training strategy for attentionbased encoder-decoder acoustic-to-word end-to-end systems. Accuracy of end-to-end systems has greatly improved thanks to careful tuning of model structure and the introduction of novel training strategies to stabilize training. For example, multi-task learning using a shared-encoder is often used to escape from bad local optima. However, multi-task learning usually relies on a linear interpolation of the losses for each sub-task and consequently, the shared-encoder is not optimized for each task. To solve the above problem, we propose a multi-task learning with augmentation strategy. We augment the training data by creating multiple copies of the original training data to suit different output targets associated with each sub-task. We use each target loss sequentially to update the parameters of the shared-encoder so as to enhance the versatility of capturing acoustic features. This strategy enables better learning of the shared-encoder as each task is trained with a dedicated loss. The parameters of the word-decoder are jointly updated via the shared-encoder when optimizing the word prediction task loss. We evaluate our proposal on various speech data sets, and show that our models achieve lower word error rates than both single-task and conventional multi-task approaches.

show abstract

Efficient Knowledge Distillation from an Ensemble of Teachers

Cited by 212 publications

References 11 publications

Direct Neuron-Wise Fusion of Cognate Neural Networks

Direct Neuron-Wise Fusion of Cognate Neural Networks

Blind Bandwidth Extension with a Non-Linear Function and Its Evaluation on Automatic Speaker Verification

Multi-task Learning with Augmentation Strategy for Acoustic-to-word Attention-based Encoder-decoder Speech Recognition

Contact Info

Product

Resources

About