Overcoming Catastrophic Forgetting by Incremental Moment Matching

Lee, Sang‐Woo; Kim, Jin-Hwa; Jun, Jae Bok; Ha, Jung-Woo; Zhang, Byoung-Tak

doi:10.48550/arxiv.1703.08475

Cited by 35 publications

(41 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, most of the existing methods are proposed to address the single image classification problem, hence not readily applicable to object detection. Our method falls within the context of the regularization-based learning approach [13,10,14]. For example, when learning incrementally, ILDVQ [13] preserves feature representation of the network on older tasks by using a less-forgetting loss, where response targets are computed using data from the current task.…”

Section: Related Workmentioning

confidence: 99%

“…As a result, networks trained from few-shot tasks tend to be less stable and forget previously seen tasks more severely. A common approach Incremental Moment Matching (mean-IMM) [10] proposes to prevent catastrophic forgetting by averaging model weights of two sequential tasks, by implicitly assuming that the obtained optimum at each incremental step is flat and overlaps with each other. However, we argue that such assumption can only hold for data-sufficient scenarios but not for few-shot.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards Generalized and Incremental Few-Shot Object Detection

Li¹,

Zhu²,

Ma³

et al. 2021

Preprint

View full text Add to dashboard Cite

Real-world object detection is highly desired to be equipped with the learning expandability that can enlarge its detection classes incrementally. Moreover, such learning from only few annotated training samples further adds the flexibility for the object detector, which is highly expected in many applications such as autonomous driving, robotics, etc. However, such sequential learning scenario with few-shot training samples generally causes catastrophic forgetting and dramatic overfitting. In this paper, to address the above incremental few-shot learning issues, a novel Incremental Few-Shot Object Detection (iFSOD) method is proposed to enable the effective continual learning from few-shot samples. Specifically, a Double-Branch Framework (DBF) is proposed to decouple the feature representation of base and novel (few-shot) class, which facilitates both the old-knowledge retention and new-class adaption simultaneously. Furthermore, a progressive model updating rule is carried out to preserve the long-term memory on old classes effectively when adapt to sequential new classes. Moreover, an inter-task class separation loss is proposed to extend the decision region of new-coming classes for better feature discrimination. We conduct experiments on both Pascal VOC and MS-COCO, which demonstrate that our method can effectively solve the problem of incremental few-shot detection and significantly improve the detection accuracy on both base and novel classes.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Towards Generalized and Incremental Few-Shot Object Detection

Li¹,

Zhu²,

Ma³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Rather than focusing on the weights in all layers of CNNs, LFL [32] restricts the drastic changes in the learned parameters in the final hidden activations to preserve the previously learned input-output mappings and maintain the decision boundaries. IMM [33] progressively matches the Gaussian posterior distribution of the CNNs trained on the old and new tasks and uses various transfer learning techniques to render the Gaussian distribution smooth and reasonable.…”

Section: A Information Retrospectionmentioning

confidence: 99%

Reviewing continual learning from the perspective of human-level intelligence

Chang¹,

Li²,

Peng³

et al. 2021

Preprint

View full text Add to dashboard Cite

Humans continual learning (CL) ability is closely related to Stability Versus Plasticity Dilemma that describes how humans achieve ongoing learning capacity and preservation for learned information. The notion of CL has always been present in artificial intelligence (AI) since its births. This paper proposes a comprehensive review on CL. Different from previous reviews that mainly focus on the catastrophic forgetting phenomenon in CL, this paper surveys CL from a more macroscopic perspective based on the Stability Versus Plasticity mechansim. Analogous to biological counterpart, "smart" AI agents are supposed to i) remember previously learned information (information retrospection); ii) infer on new information continuously (information prospection:); iii) transfer useful information (information transfer), to achieve high-level CL. According to the taxonomy, evaluation metrics, algorithms, applications as well as some open issues are then introduced. Our main contributions concern i) recheck CL from the level of artificial general intelligence; ii) provide a detailed and extensive overview on CL topics; iii) present some novel ideas on the potential development of CL.

show abstract

“…Regularization-based techniques such as elastic weight consolidation (EWC) [18], synaptic intelligence (SI) [52] and incremental moment matching (IMM) [24], modify model parameters in such a way that preserves important weights for previous tasks while finding less sensitive weights to accommodate new tasks. Despite the popularity of these methods, they tend not to perform well in class-incremental settings since they were originally introduced for task-incremental learning [48].…”

Section: Continual Learningmentioning

confidence: 99%

“…On the other hand, for Class-Incremental test settings, the model is estimating 𝑃 (𝑦 𝑗 = 𝑘 |x 𝑗 ) since it is only presented with the unknown x 𝑗 without any additional information to which task the unknown samples belong to. This in turn makes class-incremental settings significantly more challenging than taskincremental settings and explains why most of the work published in the field assumes that 𝑡 𝑖 is presented during testing [24,26,36]. To elicit the intuition behind our approach, we first break the problem of class-incremental learning into a more granular task-agnostic formulation, namely, instead of assuming that the model is presented with a stream of tasks containing multiple classes, we assume that the model is presented with a stream of independent classes.…”

Section: Introductionmentioning

confidence: 99%

Latent-Insensitive autoencoders for Anomaly Detection

Battikh¹,

Lenskiy²

2021

Preprint

View full text Add to dashboard Cite

Reconstruction-based approaches to anomaly detection tend to fall short when applied to complex datasets with target classes that possess high interclass variance [11,33]. Similar to the idea of self-taught learning [34] used in transfer learning, many domains are rich with similar unlabeled datasets that could be leveraged as a proxy for out-of-distribution samples. In this paper we introduce Latent-Insensitive Autoencoder (LIS-AE) where unlabeled data from a similar domain is utilized as negative examples to shape the latent layer (bottleneck) of a regular autoencoder such that it is only capable of reconstructing one task. Since the underlying goal of LIS-AE is to only reconstruct in-distribution samples, this makes it naturally applicable in the domain of class-incremental learning. We treat class-incremental learning as multiple anomaly detection tasks by adding a different latent layer for each class and use other available classes in task as negative examples to shape each latent layer. We test our model in multiple anomaly detection and class-incremental settings presenting quantitative and qualitative analysis showcasing the accuracy and the flexibility of our model for both anomaly detection and class-incremental learning.

show abstract

Overcoming Catastrophic Forgetting by Incremental Moment Matching

Cited by 35 publications

References 10 publications

Towards Generalized and Incremental Few-Shot Object Detection

Towards Generalized and Incremental Few-Shot Object Detection

Reviewing continual learning from the perspective of human-level intelligence

Latent-Insensitive autoencoders for Anomaly Detection

Contact Info

Product

Resources

About