Balancing Stability and Plasticity Through Advanced Null Space in Continual Learning

Kong, Yajing; Liu, Liu; Wang, Zhen; Tao, Dacheng

doi:10.1007/978-3-031-19809-0_13

Cited by 11 publications

(3 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Advanced Null Space (AdNS) is the most similar work to ours, which addresses the overfitting problem by a similar regularization construction in a multiclass continual learning scenario [23]. However, their approach differs in that there is no general rule for setting the regularization strength, and each task has a separate non-orthogonalized multiclass classifier, possibly degrading the performance on the base classes.…”

Section: Related Workmentioning

confidence: 99%

“…The novel class is then added using the imprinting technique [30]. This technique computes the features of the available novel training examples for the last layer ϕ L (•) and uses the arithmetic average as the new weight vector: [13] 52.1±11.3 -2.5±01.9 77.8±11.0 00.0±00.0 91.7±07.2 00.0±00.0 60.6±16.9 00.0±00.0 66.7±37.2 -2.4±01.8 59.0±07.9 00.0±00.0 100.0±00.0 00.0±00.0 72.8±22.4 -0.7±01.5 AdNS [23] 100.0±00.0 -88.9±00.0 93.8±04.2 00.0±00.0 100.0±00.0 -3.2±00.0 100.0±00.0 -96.6±00.0 100.0±00.0 -6.0±04.1 76.9±07.7 -2.2±00.0 100.0±00.0 -3.6±00.0 96.1±08.2 -32.5±42.9 ProtoNet [2] 37.5±06.2 00.0±00.0 47.2±12.7 00.0±00.0 86.1±09.6 00.0±00.0 60.6±18.9 00.0±00.0 66.7±41.6 -3.6±00.0 56.4±04.4 00.0±00.0 98.1±03.2 03.6±00.0 64.7±25.5 00.0±02.0 iCaRL [8] 35.4±13.0 -3.7±03.7 80.6±04.8 00.0±00.0 88.9±04.8 00.0±00.0 72.7±00.0 00.0±00.0 86.7±11.5 -9.5±05.5 51.3±08.9 -7.2±01.3 100.0±00.0 -36.9±02.1 73.6±22.6 -8.2±12.7 Finetune 58.3±03.6 -3.7±00.0 52.8±09.6 00.0±00.0 100.0±00.0 00.0±00.0 75.8±10.5 00.0±00.0 80.0±34.6 -8.3±05.5 61.5±07.7 00.0±00.0 100.0±00.0 -0.0±03.6 75.5±21.9 -1.7±03.7…”

Section: Null-space Initialization Of the Output Layermentioning

confidence: 99%

See 1 more Smart Citation

Incremental one-class learning using regularized null-space training for industrial defect detection

Hermann,

Umlauf,

Goldlücke

et al. 2024

Sixteenth International Conference on Machine Vision (ICMV 2023)

View full text Add to dashboard Cite

One-class incremental learning is a special case of class-incremental learning, where only a single novel class is incrementally added to an existing classifier instead of multiple classes. This case is relevant in industrial defect detection scenarios, where novel defects usually appear during operation. Existing rolled-out classifiers must be updated incrementally in this scenario with only a few novel examples. In addition, it is often required that the base classifier must not be altered due to approval and warranty restrictions. While simple finetuning often gives the best performance across old and new classes, it comes with the drawback of potentially losing performance on the base classes (catastrophic forgetting [1]). Simple prototype approaches [2] work without changing existing weights and perform very well when the classes are well separated but fail dramatically when not. In theory, null-space training (NSCL) [3] should retain the basis classifier entirely, as parameter updates are restricted to the null space of the network with respect to existing classes. However, as we show, this technique promotes overfitting in the case of one-class incremental learning. In our experiments, we found that unconstrained weight growth in null space is the underlying issue, leading us to propose a regularization term (R-NSCL) that penalizes the magnitude of amplification. The regularization term is added to the standard classification loss and stabilizes null-space training in the one-class scenario by counteracting overfitting. We test the method's capabilities on two industrial datasets, namely AITEX and MVTec, and compare the performance to state-of-the-art algorithms for class-incremental learning.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Null-space Initialization Of the Output Layermentioning

confidence: 99%

Incremental one-class learning using regularized null-space training for industrial defect detection

Hermann,

Umlauf,

Goldlücke

et al. 2024

Sixteenth International Conference on Machine Vision (ICMV 2023)

View full text Add to dashboard Cite

show abstract

“…Continual Learning (CL) studies the problem of learning new knowledge incrementally while preserving previously learned knowledge. Following [10], CL methods can be divided into three groups: rehearsal-based methods [30,44,36,3,37,11], regularizationbased methods [18,20,22,19,42], and parameter isolation methods [31,32,28]. However, the majority of existing CL methods focus on supervised settings while Continual Self-Supervised Continual Learning (CSSL) is surprisingly under-investigated.…”

Section: Continual Learningmentioning

confidence: 99%

Adaptive Self-Supervised Continual Learning

Wu,

Wang,

Liu

2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

Continual Learning (CL) studies the problem of developing a robust model that can learn new tasks while retaining previously learned knowledge. However, the current CL methods exclusively focus on data with annotations, disregarding that unlabelled data is the mainstream in real-world applications. To close this research gap, this study concentrates on continual self-supervised learning, which is plagued by challenges of memory over-fitting and class imbalance. Besides, these challenges are exacerbated throughout incremental training. Aimed at addressing these challenges from both loss and data perspectives, we introduce a framework, Adaptive Self-supervised Continual Learning (ASCL). Specifically, we devise an Adaptive Sharpness-Aware Minimization (ASAM) module responsible for identifying flatter local minima in the loss landscape with a smaller memory over-fitting risk. Additionally, we design an Adaptive Memory Enhancement (AME) module responsible for rebalancing self-supervised loss with new and old tasks from a data perspective. Finally, the adaptive mechanisms in AME and ASAM modules dynamically adjust the loss landscape sharpness and memory enhancement strength with the feedback of intermediate training results. The results of our extensive experiments demonstrate the state-of-the-art performance of our methods in continual self-supervised learning scenarios across multiple datasets.

show abstract

Trust-Region Adaptive Frequency for Online Continual Learning

et al. 2023

Self Cite

View full text Add to dashboard Cite

In the paradigm of online continual learning, one neural network is exposed to a sequence of tasks, where the data arrive in an online fashion and previously seen data are not accessible. Such online fashion causes insufficient learning and severe forgetting on past tasks issues, preventing a good stability-plasticity trade-off, where ideally the network is expected to have high plasticity to adapt to new tasks well and have the stability to prevent forgetting on old tasks simultaneously. To solve these issues, we propose a trust-region adaptive frequency approach, which alternates between standard-process and intra-process updates. Specifically, the standard-process replays data stored in a coreset and interleaves the data with current data, and the intra-process updates the network parameters based on the coreset. Furthermore, to improve the unsatisfactory performance stemming from online fashion, the frequency of the intra-process is adjusted based on a trust region, which is measured by the confidence score of current data. During the intra-process, we distill the dark knowledge to retain useful learned knowledge. Moreover, to store more representative data in the coreset, a confidence-based coreset selection is presented in an online manner. The experimental results on standard benchmarks show that the proposed method significantly outperforms state-of-art continual learning algorithms.

show abstract

Balancing Stability and Plasticity Through Advanced Null Space in Continual Learning

Cited by 11 publications

References 30 publications

Incremental one-class learning using regularized null-space training for industrial defect detection

Incremental one-class learning using regularized null-space training for industrial defect detection

Adaptive Self-Supervised Continual Learning

Trust-Region Adaptive Frequency for Online Continual Learning

Contact Info

Product

Resources

About