Online Continual Learning with Contrastive Vision Transformer

Wang, Zhen; Liu, Liu; Kong, Yajing; Guo, Jiaxian; Tao, Dacheng

doi:10.1007/978-3-031-20044-1_36

Cited by 21 publications

(8 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It learns representations by contrasting positive and negative samples and has proven effective in various tasks, including image classification [17,41,42], object detection [43,44], and natural language processing [45]. Consequently, contrastive representation learning has garnered substantial attention in recent years within the lifelong or continual learning literature [19,20,[46][47][48][49][50]. By harnessing the principles of contrastive learning, L3 models can acquire representations that capture both task-specific information and general features.…”

Section: Memory-replay-based L3 Methodsmentioning

confidence: 99%

CL3: Generalization of Contrastive Loss for Lifelong Learning

Roy,

Simon,

Moghadam

et al. 2023

J. Imaging

View full text Add to dashboard Cite

Lifelong learning portrays learning gradually in nonstationary environments and emulates the process of human learning, which is efficient, robust, and able to learn new concepts incrementally from sequential experience. To equip neural networks with such a capability, one needs to overcome the problem of catastrophic forgetting, the phenomenon of forgetting past knowledge while learning new concepts. In this work, we propose a novel knowledge distillation algorithm that makes use of contrastive learning to help a neural network to preserve its past knowledge while learning from a series of tasks. Our proposed generalized form of contrastive distillation strategy tackles catastrophic forgetting of old knowledge, and minimizes semantic drift by maintaining a similar embedding space, as well as ensures compactness in feature distribution to accommodate novel tasks in a current model. Our comprehensive study shows that our method achieves improved performances in the challenging class-incremental, task-incremental, and domain-incremental learning for supervised scenarios.

show abstract

Section: Memory-replay-based L3 Methodsmentioning

confidence: 99%

CL3: Generalization of Contrastive Loss for Lifelong Learning

Roy,

Simon,

Moghadam

et al. 2023

J. Imaging

View full text Add to dashboard Cite

show abstract

“…Such assignment is a challenge that resides at the core of all the aforementioned dynamic learning paradigms. In particular, the focus here is on case study examples where new exceptions and categories are learned in real time so that mitigation of the phenomenon that has been identified as 'catastrophic forgetting' [10,[13][14][15][16][17][18] is considered.…”

Section: Literature Studymentioning

confidence: 99%

“…Distillation loss on the old classes and cross-entropy loss on the new class are jointly optimised, which in turn gives good performance on the classification task of the new as well as old classes. Continual learning methodologies have been classified into three groups in [16]. They are expansion-based, regularisation-based and rehearsal-based methods.…”

Section: Incremental Learning In Manufacturingmentioning

confidence: 99%

A Survey of Incremental Deep Learning for Defect Detection in Manufacturing

Mohandas,

Southern,

O’Connell

et al. 2024

BDCC

View full text Add to dashboard Cite

Deep learning based visual cognition has greatly improved the accuracy of defect detection, reducing processing times and increasing product throughput across a variety of manufacturing use cases. There is however a continuing need for rigorous procedures to dynamically update model-based detection methods that use sequential streaming during the training phase. This paper reviews how new process, training or validation information is rigorously incorporated in real time when detection exceptions arise during inspection. In particular, consideration is given to how new tasks, classes or decision pathways are added to existing models or datasets in a controlled fashion. An analysis of studies from the incremental learning literature is presented, where the emphasis is on the mitigation of process complexity challenges such as, catastrophic forgetting. Further, practical implementation issues that are known to affect the complexity of deep learning model architecture, including memory allocation for incoming sequential data or incremental learning accuracy, is considered. The paper highlights case study results and methods that have been used to successfully mitigate such real-time manufacturing challenges.

show abstract

“…Contrastive learning [17,29,37,8,32,20,28] has already demonstrated impressive visual representation learning capabilities. In selfsupervised learning, the idea of "contrastive" is well reflected in the pretext task Instance Discrimination [37].…”

Section: Contrastive Learningmentioning

confidence: 99%

Contrastive Learning with Diverse Samples

Wu,

Liu

2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

Unsupervised visual representation learning has gained much attention from the computer vision community because of the recent contrastive learning achievements. Current work mainly adopts instance discrimination as the pretext task, which treats every single instance as a different class (negative) and uses a collection of data augmentation techniques to generate more examples (positive) for each class. The idea is straightforward and efficient but will generally cause similar instances to be classified into different classes. Such problem has been defined as “class collision” in some previous works and is shown to hurt the representation ability. Motivated by this observation, we present a solution to address this issue by filtering similar negative examples from each mini-batch. Concretely, we model the problem as a Determinantal Point Process (DPP) so that similar instances can be filtered stochastically, and diverse samples are expected to be sampled for contrastive training. Besides, we further introduce a priority term for each instance, which indicates the hardness of its positives, so that instances with more hard positives are more likely to be sampled for contributing to the optimization. Our sampling can be efficiently implemented in a feed-forward manner and further accelerated by our encouraged complement DPP. Extensive experimental results demonstrate our priority over the standard setup of contrastive learning.

show abstract

Online Continual Learning with Contrastive Vision Transformer

Cited by 21 publications

References 58 publications

CL3: Generalization of Contrastive Loss for Lifelong Learning

CL3: Generalization of Contrastive Loss for Lifelong Learning

A Survey of Incremental Deep Learning for Defect Detection in Manufacturing

Contrastive Learning with Diverse Samples

Contact Info

Product

Resources

About