Combining Curriculum Learning and Knowledge Distillation for Dialogue Generation

Zhu, Qiuming; Chen, Xiuying; Wu, Pengfei; Liu, JunFei; Zhao, Dongyan

doi:10.18653/v1/2021.findings-emnlp.111

Cited by 11 publications

(10 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Knowledge Distillation. Knowledge distillation (KD) has been actively studied for model compression in various fields [5,11,17,37,48,55]. KD transfers the knowledge captured by a teacher model through large capacity into a lightweight student model, significantly lowering the inference cost while maintaining comparable performance.…”

Section: Related Workmentioning

confidence: 99%

“…On the other hand, self-paced learning [21] makes the curriculum dynamically adjusted during the training, usually based on training loss [21] or performance on the validation set [6,49]. The easy-tohard learning has been applied to KD to improve the distillation efficacy in computer vision [14,39] and natural language processing [52,55]. [3,14,39] exploit the teacher's optimization route to form a curriculum for the student, [52] gradually includes an increasing number of fine-grained document pairs during the training.…”

Section: Related Workmentioning

confidence: 99%

“…In this work, we show the necessity and difficulty of distilling the ensemble of heterogeneous teachers and apply the easy-to-hard learning to cope with the problem. Further, the prior KD works with the easy-to-hard learning focus on classification [14,31,39] or rely on domain-specific features [55], which makes them hard to apply to the ranking problem and recommender system. Our work provides a solution tailored to compress ranking models by distilling an easy-to-hard sequence of ranking knowledge considering the student's learning state.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Distillation from Heterogeneous Models for Top-K Recommendation

Kang

Kweon

Lee

et al. 2023

Proceedings of the ACM Web Conference 2023

View full text Add to dashboard Cite

Recent recommender systems have shown remarkable performance by using an ensemble of heterogeneous models. However, it is exceedingly costly because it requires resources and inference latency proportional to the number of models, which remains the bottleneck for production. Our work aims to transfer the ensemble knowledge of heterogeneous teachers to a lightweight student model using knowledge distillation (KD), to reduce the huge inference costs while retaining high accuracy. Through an empirical study, we find that the efficacy of distillation severely drops when transferring knowledge from heterogeneous teachers. Nevertheless, we show that an important signal to ease the difficulty can be obtained from the teacher's training trajectory. This paper proposes a new KD framework, named HetComp, that guides the student model by transferring easy-to-hard sequences of knowledge generated from the teachers' trajectories. To provide guidance according to the student's learning state, HetComp uses dynamic knowledge construction to provide progressively difficult ranking knowledge and adaptive knowledge transfer to gradually transfer finer-grained ranking information. Our comprehensive experiments show that HetComp significantly improves the distillation quality and the generalization of the student model. CCS CONCEPTS• Information systems → Retrieval models and ranking; Recommender systems; Retrieval efficiency.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Distillation from Heterogeneous Models for Top-K Recommendation

Kang

Kweon

Lee

et al. 2023

Proceedings of the ACM Web Conference 2023

View full text Add to dashboard Cite

show abstract

“…Classical training algorithms sample instances from the corpus according to the static uniform distribution. Curriculum learning adopts dynamic data sampling strategy during training (Zhu et al, 2021;Guo et al, 2020;Qian et al, 2020). For example, it imitates taking well-designed easy-to-hard training courses, where "easy" instances are more likely to be sampled at early training stage, and "hard" instances are with higher sampling probabilities at late training stage.…”

Section: Mixed Sequence Distillationmentioning

confidence: 99%

A Self-Paced Mixed Distillation Method for Non-Autoregressive Generation

Qi¹,

Gong²,

Shen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Non-Autoregressive generation is a sequence generation paradigm, which removes the dependency between target tokens. It could efficiently reduce the text generation latency with parallel decoding in place of tokenby-token sequential decoding.However, due to the known multi-modality problem, Non-Autoregressive (NAR) models significantly under-perform Auto-regressive (AR) models on various language generation tasks. Among the NAR models, BANG is the first large-scale pre-training model on English un-labeled raw text corpus. It considers different generation paradigms as its pre-training tasks including Auto-regressive (AR), Non-Autoregressive (NAR), and semi-Non-Autoregressive (semi-NAR) information flow with multi-stream strategy. It achieves state-of-the-art performance without any distillation techniques. However, AR distillation has been shown to be a very effective solution for improving NAR performance. In this paper, we propose a novel self-paced mixed distillation method to further improve the generation quality of BANG. Firstly, we propose the mixed distillation strategy based on the AR stream knowledge. Secondly, we encourage the model to focus on the samples with the same modality by self-paced learning. The proposed self-paced mixed distillation algorithm improves the generation quality and has no influence on the inference latency. We carry out extensive experiments on summarization and question generation tasks to validate the effectiveness. To further illustrate the commercial value of our approach, we conduct experiments on three generation tasks in real-world advertisements applications. Experimental results on commercial data show the effectiveness of the proposed model. Compared with BANG, it achieves significant BLEU score improvement. On the other hand, compared with * Work is done during internship at Microsoft Research Asia.† Corresponding Author.auto-regressive generation method, it achieves more than 7x speedup. We will make our code publicly available.

show abstract

“…When there are several difficult samples, such as loud noise situations, the joint framework of robust training and compression is a popular approach in various other domains. Existing studies in the fields of computer vision and natural language processing have proposed joint frameworks for combining curriculum learning and knowledge distillation methods [28], [29]. However, studies on joint optimization methods for noise robustness are scarce in the field of acoustic modeling.…”

Section: Introductionmentioning

confidence: 99%

Joint Framework of Curriculum Learning and Knowledge Distillation for Noise-Robust and Small-Footprint Keyword Spotting

Lim,

Baek

2023

IEEE Access

View full text Add to dashboard Cite

Spoken keyword spotting has been widely used to facilitate an always-on voice interface in consumer electronics, owing to its simplicity and low latency. Small-footprint keyword spotting based on tiny convolutional neural networks can be implemented on resource-constrained but energy-efficient microcontrollers in real time. However, it is difficult for tiny neural networks to learn the noise-robustness properties essential for successful voice interfaces. To overcome this problem, this study proposes a joint framework of curriculum learning and knowledge distillation for noise-robust small-footprint keyword spotting. The proposed joint framework applies noise mixture curriculum learning to a network that is sufficiently large for learning various noise situations. Subsequently, knowledge distillation is applied to compress the large network into a network small enough to be onboard microcontrollers. To enhance the effectiveness of the joint framework, we propose curriculum learning using a new noise mixture strategy and knowledge distillation with an effective ensemble of neural network snapshots for each curriculum stage. These proposed methods enable the effective learning of noisy situations by large networks and the transfer of noise robustness to small networks. The effectiveness of our joint framework is illustrated using the Google Speech Commands dataset with noise mixtures from various public noise datasets. Our joint framework results in superior performance in noisy situations compared with state-of-the-art noise-robust keyword spotting methods. Therefore, the proposed framework significantly improves the usability of voice interfaces in consumer electronics.

show abstract

Combining Curriculum Learning and Knowledge Distillation for Dialogue Generation

Cited by 11 publications

References 33 publications

Distillation from Heterogeneous Models for Top-K Recommendation

Distillation from Heterogeneous Models for Top-K Recommendation

A Self-Paced Mixed Distillation Method for Non-Autoregressive Generation

Joint Framework of Curriculum Learning and Knowledge Distillation for Noise-Robust and Small-Footprint Keyword Spotting

Contact Info

Product

Resources

About