Continual Learning With Extended Kronecker-Factored Approximate Curvature

Lee, Janghyeon; Hong, Hyeong Gwon; Joo, Donggyu; Kim, Junmo

doi:10.1109/cvpr42600.2020.00902

Cited by 34 publications

(14 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ritter et al [27] updated a quadratic penalty term in the loss function for every task by the block-diagonal Kronecker factored approximation of the Hessian matrix for considering the intra-layer parameter interaction simultaneously. Lee et al [28] pointed out that the current usage of the Hessian matrix approximation as the curvature of the quadratic penalty function is not effective for networks containing batch normalization layers. To resolve this issue, a Hessian approximation method, which considers the effects of batch normalization layers was proposed.…”

Section: Regularizing Loss Functionmentioning

confidence: 99%

Recent Advances of Continual Learning in Computer Vision: An Overview

Qu,

Rahmani,

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Section: Regularizing Loss Functionmentioning

confidence: 99%

Recent Advances of Continual Learning in Computer Vision: An Overview

Qu,

Rahmani,

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…After that, several normalization layers devised for various computer vision tasks have been proposed with their respective advantage (Ulyanov et al, 2016;Wu & He, 2018;Hoffer et al, 2017). Recently, appropriate use of BN in more diverse tasks, such as meta-learng (Bronskill et al, 2020), task-incremental learning (Lee et al, 2020) and online class-incremental learning (Anonymous, 2022), have been discussed.…”

Section: Related Workmentioning

confidence: 99%

Task-Balanced Batch Normalization for Exemplar-based Class-Incremental Learning

Cha¹,

Cho²,

Hwang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Batch Normalization (BN) is an essential layer for training neural network models in various computer vision tasks. It has been widely used in continual learning scenarios with little discussion, but we find that BN should be carefully applied, particularly for the exemplar memory based class incremental learning (CIL). We first analyze that the empirical mean and variance obtained for normalization in a BN layer become highly biased toward the current task. To tackle its significant problems in training and test phases, we propose Task-Balanced Batch Normalization (TBBN). Given each mini-batch imbalanced between the current and previous tasks, TBBN first reshapes and repeats the batch, calculating near task-balanced mean and variance. Second, we show that when the affine transformation parameters of BN are learned from a reshaped feature map, they become less-biased toward the current task. Based on our extensive CIL experiments with CIFAR-100 and ImageNet-100 datasets, we demonstrate that our TBBN is easily applicable to most of existing exemplar-based CIL algorithms, improving their performance by decreasing the forgetting on the previous tasks.

show abstract

“…This is because sharp task-specific minima lead to overspecialization to a particular task and consequently to a forgetting of all other tasks. Weights constraints as EWC [38] or second-order optimization [42] have similar motivations. SAM estimates the worst closest parameters during a first forward/backward pass, and then optimizes the loss w.r.t.…”

Section: A3 Novel Continual Training Proceduresmentioning

confidence: 99%

DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion

Douillard¹,

Ramé²,

Couairon³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep network architectures struggle to continually learn new tasks without forgetting the previous tasks. A recent trend indicates that dynamic architectures based on an expansion of the parameters can reduce catastrophic forgetting efficiently in continual learning. However, existing approaches often require a task identifier at test-time, need complex tuning to balance the growing number of parameters, and barely share any information across tasks. As a result, they struggle to scale to a large number of tasks without significant overhead. In this paper, we propose a transformer architecture based on a dedicated encoder/decoder framework. Critically, the encoder and decoder are shared among all tasks. Through a dynamic expansion of special tokens, we specialize each forward of our decoder network on a task distribution. Our strategy scales to a large number of tasks while having negligible memory and time overheads due to strict control of the parameters expansion. Moreover, this efficient strategy doesn't need any hyperparameter tuning to control the network's expansion. Our model reaches excellent results on CIFAR100 and state-of-the-art performances on the largescale ImageNet100 and ImageNet1000 while having less parameters than concurrent dynamic frameworks. Code will be released 1 .

show abstract

Continual Learning With Extended Kronecker-Factored Approximate Curvature

Cited by 34 publications

References 15 publications

Recent Advances of Continual Learning in Computer Vision: An Overview

Recent Advances of Continual Learning in Computer Vision: An Overview

Task-Balanced Batch Normalization for Exemplar-based Class-Incremental Learning

DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion

Contact Info

Product

Resources

About