Class-incremental Learning via Deep Model Consolidation

Abstract

Deep neural networks (DNNs) often suffer from "catastrophic forgetting" during incremental learning (IL) -an abrupt degradation of performance on the original set of classes when the training objective is adapted to a newly added set of classes. Existing IL approaches tend to produce a model that is biased towards either the old classes or new classes, unless with the help of exemplars of the old data. To address this issue, we propose a classincremental learning paradigm called Deep Model Consolidation (DMC), which works well even when the original training data is not available. The idea is to first train a separate model only for the new classes, and then combine the two individual models trained on data of two distinct set of classes (old classes and new classes) via a novel double distillation training objective. The two existing models are consolidated by exploiting publicly available unlabeled auxiliary data. This overcomes the potential difficulties due to unavailability of original training data. Compared to the state-of-the-art techniques, DMC demonstrates significantly better performance in image classification (CIFAR-100 and CUB-200) and object detection (PASCAL VOC 2007) in the single-headed IL setting.

Knowledge distillation for incremental learning in semantic segmentation

“…In [6] the current model distills knowledge from all previous model snapshots, of which a pruned version is saved. Deep Model Consolidation (DMC) [40] proposes the idea to train a separate model for the new classes, and then combine the two models (for old and new data, respectively) via double distillation objective. The two models are consolidated via publicly available unlabeled auxiliary data.…”

iTAML: An Incremental Task-Agnostic Meta-learning Approach

“…Comparison with existing methods: We extensively compare iTAML with several popular incremental learning methods, including Elastic Weight Consolidation [13], Riemannian Walk (RWalk) [3], Learning without Forgetting (LwF) [15], Synaptic Intelligence (SI) [31], Memory Aware Synapses (MAS) [1], Deep Model Consolidation (DMC) [32], Incremental Classifier and Representation Learning (iCARL) [23], Random Path Selection network (RPS-net) [21] and Bias Correction Method (BiC) [30]. We also compare against Fixed representations (FixedRep) and Fine tuning (FineTune).…”

FFNB: Forgetting-Free Neural Blocks for Deep Continual Visual Learning

“…Other models seek to leverage extra knowledge including unlabeled data [28], [72] (which are independent from the targeted tasks) or biases (due to imbalanced distributions) between previous and current tasks to further enhance generalization [15], [63].…”