M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning

Zhou, Peng; Mai, Long; Zhang, Jianming; Xu, Ning; Wu, Zuxuan; Davis, Larry S.

doi:10.48550/arxiv.1904.01769

Cited by 9 publications

(11 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Knowledge distillation provides an effective way to preserve and transfer learned knowledge without catastrophic forgetting. Recently, an increasing number of KD variants, which are based on the lifelong learning, have been developed (Jang et al, 2019;Flennerhag et al, 2019;Peng et al, 2019b;Liu et al, 2019d;Lee et al, 2019b;Zhai et al, 2019;Zhou et al, 2019c;Shmelkov et al, 2017;Li and Hoiem, 2017). The methods proposed in (Jang et al, 2019;Peng et al, 2019b;Liu et al, 2019d;Flennerhag et al, 2019) adopt meta-learning.…”

Section: Lifelong Distillationmentioning

confidence: 99%

“…The teacher knowledge obtained from the image modalities and semantic information are preserved and transferred. Moreover, to address the problem of catastrophic forgetting in lifelong learning, global distillation (Lee et al, 2019b), multi-model distillation (Zhou et al, 2019c), knowledge distillation-based lifelong GAN (Zhai et al, 2019) and the other KD-based methods (Li and Hoiem, 2017;Shmelkov et al, 2017) have been developed to extract the learned knowledge and teach the student network on new tasks.…”

Section: Lifelong Distillationmentioning

confidence: 99%

See 1 more Smart Citation

Knowledge Distillation: A Survey

Gou,

Yu,

Maybank

et al. 2020

Preprint

View full text Add to dashboard Cite

In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. The great success of deep learning is mainly due to its scalability to encode large-scale data and to maneuver billions of model parameters. However, it is a challenge to deploy these cumbersome deep models on devices with limited resources, e.g., mobile phones and embedded devices, not only because of the high computational complexity but also the large storage requirements. To this end, a variety of model compression and acceleration techniques have been developed. As a representative type of model compression and acceleration, knowledge distillation effectively learns a small student model from a large teacher model. It has received rapid increasing attention from the community. This paper provides a comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, distillation algorithms and applications. Furthermore, challenges in knowledge distillation are briefly reviewed and comments on future research are discussed and forwarded.

show abstract

Section: Lifelong Distillationmentioning

confidence: 99%

Section: Lifelong Distillationmentioning

confidence: 99%

Knowledge Distillation: A Survey

Gou,

Yu,

Maybank

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…The LwF method does not require any old data to be stored and uses KD as an additional regularization term on the loss function to force the new model to follow the behavior of the old model on old tasks. (Zhou et al, 2019) proposed a multi-model distillation method called M2KD which directly matches the category outputs of the current model with those of the corresponding old models. Mask based pruning is used to compress the old models in M2KD.…”

Section: Kd Based Incremental Learning Methodsmentioning

confidence: 99%

Faster ILOD: Incremental Learning for Object Detectors based on Faster RCNN

Peng

Zhao

Lovell

2020

Preprint

View full text Add to dashboard Cite

The human vision and perception system is inherently incremental where new knowledge is continually learned over time whilst existing knowledge is retained. On the other hand, deep learning networks are ill-equipped for incremental learning. When a well-trained network is adapted to new categories, its performance on the old categories will dramatically degrade. To address this problem, incremental learning methods have been explored to preserve the old knowledge of deep learning models. However, the state-of-the-art incremental object detector employs an external fixed region proposal method that increases overall computation time and reduces accuracy compared to object detectors such as Faster RCNN that use trainable Region Proposal Networks (RPNs). The purpose of this paper is to design an efficient end-to-end incremental object detector using knowledge distillation for object detectors based on RPNs. We first evaluate and analyze the performance of RPN-based detector with classic distillation towards incremental detection tasks. Then, we introduce multi-network adaptive distillation that properly retains knowledge from the old categories when fine-turning the model for new task. Experiments on the benchmark datasets, PASCAL VOC and COCO, demonstrate that the proposed incremental detector is more accurate as well as being 13 times faster than the baseline detector.

show abstract

“…Li et al first applied KD for incremental learning and built an incremental classifier called Learning without Forgetting (LwF) [19]. Research in this type of method aims to acquire the old task information through various distillation approaches [39], [40]. Architectural Strategies solve the catastrophic forgetting problem by designing a unique sub-network for each incremental task.…”

Section: A Incremental Classificationmentioning

confidence: 99%

DIODE: Dilatable Incremental Object Detection

Peng¹,

Zhao²,

Maksoud³

et al. 2021

Preprint

View full text Add to dashboard Cite

To accommodate rapid changes in the real world, the cognition system of humans is capable of continually learning concepts. On the contrary, conventional deep learning models lack this capability of preserving previously learned knowledge. When a neural network is fine-tuned to learn new tasks, its performance on previously trained tasks will significantly deteriorate. Many recent works on incremental object detection tackle this problem by introducing advanced regularization. Although these methods have shown promising results, the benefits are often short-lived after the first incremental step. Under multi-step incremental learning, the trade-off between old knowledge preserving and new task learning becomes progressively more severe. Thus, the performance of regularization-based incremental object detectors gradually decays for subsequent learning steps. In this paper, we aim to alleviate this performance decay on multi-step incremental detection tasks by proposing a dilatable incremental object detector (DIODE). For the task-shared parameters, our method adaptively penalizes the changes of important weights for previous tasks. At the same time, the structure of the model is dilated or expanded by a limited number of task-specific parameters to promote new task learning. Extensive experiments on PASCAL VOC and COCO datasets demonstrate substantial improvements over the state-of-the-art methods. Notably, compared with the state-of-the-art methods, our method achieves up to 6.0% performance improvement by increasing the number of parameters by just 1.2% for each newly learned task.

show abstract

M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning

Cited by 9 publications

References 32 publications

Knowledge Distillation: A Survey

Knowledge Distillation: A Survey

Faster ILOD: Incremental Learning for Object Detectors based on Faster RCNN

DIODE: Dilatable Incremental Object Detection

Contact Info

Product

Resources

About