Learning without Memorizing

Dhar, Prithviraj; Singh, Rajat Vikram; Peng, Kuan-Chuan; Wu, Ziyan; Chellappa, Rama

doi:10.48550/arxiv.1811.08051

Cited by 5 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [5] previous knowledge is distilled directly from the last trained model. In [39] an attention distillation loss is introduced as an information preserving penalty for the classifiers' attention maps. In [6] the current model distills knowledge from all previous model snapshots, of which a pruned version is saved.…”

Section: Related Workmentioning

confidence: 99%

Knowledge distillation for incremental learning in semantic segmentation

Michieli

Zanuttigh

2021

Computer Vision and Image Understanding

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 99%

Knowledge distillation for incremental learning in semantic segmentation

Michieli

Zanuttigh

2021

Computer Vision and Image Understanding

View full text Add to dashboard Cite

“…Deep learning models are prone to catastrophic forgetting [20,30,48], i.e., training a model with new information interferes with previously learned knowledge and typically greatly degrades performance. This phenomenon has been widely studied in image classification task and most of the current techniques fall into the following categories [10,48]: regularization approaches [5,32,73,13,36], dynamic architectures [69,64,35], parameter isolation [17,53,40] and replay-based methods [66,46,55,26]. Regularization-based approaches are by far the most widely employed and mainly come in two flavours, i.e., penalty computing and knowledge distillation [25].…”

Section: Related Workmentioning

confidence: 99%

“…Penalty computing approaches [73,32,32] protect important weights inside the models to prevent forgetting. Knowledge distillation [52,66,36,13] relies on a teacher (old) model transferring or remembering knowledge related to previous tasks to a student model which is trained to learn also additional tasks. Parameter isolation approaches [40,39] reserve a subset of weights for a specific task to avoid degradation.…”

Section: Related Workmentioning

confidence: 99%

Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations

Michieli¹,

Zanuttigh²

2021

Preprint

View full text Add to dashboard Cite

Deep neural networks suffer from the major limitation of catastrophic forgetting old tasks when learning new ones.In this paper we focus on class incremental continual learning in semantic segmentation, where new categories are made available over time while previous training data is not retained. The proposed continual learning scheme shapes the latent space to reduce forgetting whilst improving the recognition of novel classes. Our framework is driven by three novel components which we also combine on top of existing techniques effortlessly. First, prototypes matching enforces latent space consistency on old classes, constraining the encoder to produce similar latent representation for previously seen classes in the subsequent steps. Second, features sparsification allows to make room in the latent space to accommodate novel classes. Finally, contrastive learning is employed to cluster features according to their semantics while tearing apart those of different classes. Extensive evaluation on the Pascal VOC2012 and ADE20K datasets demonstrates the effectiveness of our approach, significantly outperforming state-of-the-art methods.

show abstract

“…Li et al [17] Learn Without Forgetting (LWF) by distilling the knowledge from the last model. Dhar et al [4] introduce Grad-CAM [27] in the loss function. Rebuffi et al [25] introduced exemplar set for the old data and match previous logits through distillation.…”

Section: Related Workmentioning

confidence: 99%

“…Treating the most previous model as the teacher and applying this distillation sequentially helps preserve historical information, especially when no previous exemplar set is stored, which is the protocol for prior methods [25,2,17,4]. However, the historical information will be gradually lost in this sequential pipeline as the current model must reconstruct all the prior information from the penultimate model alone.…”

Section: Multi-model Distillationmentioning

confidence: 99%

M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning

Zhou¹,

Mai²,

Zhang³

et al. 2019

Preprint

View full text Add to dashboard Cite

Incremental learning targets at achieving good performance on new categories without forgetting old ones. Knowledge distillation has been shown critical in preserving the performance on old classes. Conventional methods, however, sequentially distill knowledge only from the last model, leading to performance degradation on the old classes in later incremental learning steps. In this paper, we propose a multi-model and multi-level knowledge distillation strategy. Instead of sequentially distilling knowledge only from the last model, we directly leverage all previous model snapshots. In addition, we incorporate an auxiliary distillation to further preserve knowledge encoded at the intermediate feature levels. To make the model more memory efficient, we adapt mask based pruning to reconstruct all previous models with a small memory footprint. Experiments on standard incremental learning benchmarks show that our method preserves the knowledge on old classes better and improves the overall performance over standard distillation techniques. * This work was partially done while Peng Zhou was an intern at Adobe. Incremental stepS1 S2 S3 Knowledge distillation Knowledge distillation Incremental step S1 S2 S3 Knowledge distillation Knowledge distillation

show abstract

Learning without Memorizing

Cited by 5 publications

References 0 publications

Knowledge distillation for incremental learning in semantic segmentation

Knowledge distillation for incremental learning in semantic segmentation

Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations

M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning

Contact Info

Product

Resources

About