Revisiting Distillation and Incremental Classifier Learning

Javed, Khurram

doi:10.48550/arxiv.1807.02802

Cited by 3 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compare with the well-known rehearsal-based method iCaRL which proposes to select representative samples for old class incremental tasks and use a distillation loss to train. In our reimplementation, we set the capacity of representative set K = 2000 and use random exemplar selection rather than exemplar selection by herding since there is no substantial difference as claimed in [43]. And we train our ICLNet with the same selected representative sample set for fair comparison.…”

Section: F Compare With Rehearsal-based Methodsmentioning

confidence: 99%

Incremental Concept Learning via Online Generative Memory Recall

Dong

2019

Preprint

View full text Add to dashboard Cite

The ability to learn more and more concepts over time from incrementally arriving data is essential for the development of a life-long learning system. However, deep neural networks often suffer from forgetting previously learned concepts when continually learning new concepts, which is known as catastrophic forgetting problem. The main reason for catastrophic forgetting is that the past concept data is not available and neural weights are changed during incrementally learning new concepts. In this paper, we propose a pseudo-rehearsal based class incremental learning approach to make neural networks capable of continually learning new concepts. We use a conditional generative adversarial network to consolidate old concepts memory and recall pseudo samples during learning new concepts and a balanced online memory recall strategy is to maximally maintain old memories. And we design a comprehensible incremental concept learning network as well as a concept contrastive loss to alleviate the magnitude of neural weights change. We evaluate the proposed approach on MNIST, Fashion-MNIST and SVHN datasets and compare with other rehearsal based approaches. The extensive experiments demonstrate the effectiveness of our approach.

show abstract

Section: F Compare With Rehearsal-based Methodsmentioning

confidence: 99%

Incremental Concept Learning via Online Generative Memory Recall

Dong

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…We argue that pruning secondary parameters is sub-optimal in the case of single-head protocol Memory rehearsal and pseudo memory rehearsal. To mitigate knowledge bias toward new tasks, some methods store previous data and retrain them [5,7,8,21,22], or train generative adversarial networks (GANs) to generate and discriminate images and then learn the data distribution [28][29][30][31]. Memory rehearsal methods require additional storage to store previous data or extra model parameters to generate and discriminate data.…”

Section: Related Workmentioning

confidence: 99%

“…In the family of the single-network methods, previous works have explored the regularization methods [2,4,[16][17][18], the parameter isolation methods [19,20] and the memory rehearsal methods [5,7,8,21,22]. The regularization methods leverage a penalty term in the loss function to regularize the parameters when updating for new tasks.…”

Section: Introductionmentioning

confidence: 99%

Efficient continual learning at the edge with progressive segmented training

Venkataramanaiah

et al. 2022

Neuromorph. Comput. Eng.

View full text Add to dashboard Cite

There is an increasing need for continual learning in dynamic systems at the edge, such as self-driving vehicles, surveillance drones, and robotic systems. Such a system requires learning from the data stream, training the model to preserve previous information and adapt to a new task, and generating a single-headed vector for future inference, within a limited power budget. Different from previous continual learning algorithms with dynamic structures, this work focuses on a single network and model segmentation to mitigate catastrophic forgetting problem. Leveraging the redundant capacity of a single network, model parameters for each task are separated into two groups: one important group which is frozen to preserve current knowledge, and a secondary group to be saved (not pruned) for future learning. A fixed-size memory containing a small amount of previously seen data is further adopted to assist the training. Without additional regularization, the simple yet effective approach of Progressive Segmented Training (PST) successfully incorporates multiple tasks and achieves state-of-the-art accuracy in the single-head evaluation on the CIFAR-10 and CIFAR-100 datasets. Moreover, the segmented training significantly improves computation efficiency in continual learning and thus, enabling efficient continual learning at the edge. On Intel Stratix-10 MX FPGA, we further demonstrate the efficiency of PST with representative CNNs trained on CIFAR-10.

show abstract

“…In the family of the single-network methods, previous works have explored the regularization methods [1,4,14,18,33], the parameter isolation methods [21,22] and the memory rehearsal methods [3,5,12,20,27]. The regularization methods leverage a penalty term in the loss function to regularize the parameters when updating for new tasks.…”

Section: Introductionmentioning

confidence: 99%

Single-Net Continual Learning with Progressive Segmented Training

Charan

Liu

et al. 2019

2019 18th IEEE International Conference on Machine Learning and Applications (ICMLA)

View full text Add to dashboard Cite

There is an increasing need of continual learning in dynamic systems, such as the self-driving vehicle, the surveillance drone, and the robotic system. Such a system requires learning from the data stream, training the model to preserve previous information and adapt to a new task, and generating a single-headed vector for future inference. Different from previous approaches with dynamic structures, this work focuses on a single network and model segmentation to prevent catastrophic forgetting. Leveraging the redundant capacity of a single network, model parameters for each task are separated into two groups: one important group which is frozen to preserve current knowledge, and secondary group to be saved (not pruned) for a future learning. A fixed-size memory containing a small amount of previously seen data is further adopted to assist the training. Without additional regularization, the simple yet effective approach of PST successfully incorporates multiple tasks and achieves the state-of-the-art accuracy in the singlehead evaluation on CIFAR-10 and CIFAR-100 datasets. Moreover, the segmented training significantly improves computation efficiency in continual learning.Preprint. Under review.

show abstract

Revisiting Distillation and Incremental Classifier Learning

Cited by 3 publications

References 0 publications

Incremental Concept Learning via Online Generative Memory Recall

Incremental Concept Learning via Online Generative Memory Recall

Efficient continual learning at the edge with progressive segmented training

Single-Net Continual Learning with Progressive Segmented Training

Contact Info

Product

Resources

About