Practical Recommendations for Replay-Based Continual Learning Methods

Merlin, Gabriele; Lomonaco, Vincenzo; Cossu, Andrea; Carta, Antonio; Bacciu, Davide

doi:10.1007/978-3-031-13324-4_47

Cited by 9 publications

(8 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To solve this issue, there are currently three categories of approaches, i.e., replay methods, regularization methods, and parameter isolation methods. Replay involves periodically training on a subset of upstream task data, thereby retaining knowledge of previous tasks and balancing old and new information (Rebuffi et al 2017;Rolnick et al 2019;Liu et al 2020;Merlin et al 2022). However, storing and managing updtream task data pose challenges in terms of efficiency, particularly in the contemporary era of massive datasets (Schuhmann et al 2022;Li et al 2023).…”

Section: Fine-tuning Methodsmentioning

confidence: 99%

“…Finally, OLOR is incorporated into optimizers, thereby introducing negligible extra computational overhead. It also works well with popular optimizers such as Adam (Loshchilov and Hutter 2017;Guan 2023) and SGD (Keskar and Socher 2017), meeting specific needs under var-…”

Section: Introductionmentioning

confidence: 96%

“…Rehearsal methods (Rebuffi et al 2017;Rolnick et al 2019;Liu et al 2020;Merlin et al 2022), based on the replay mechanism, involve retraining on a subset of stored upstream samples while learning new tasks. However, this approach is quite inefficient.…”

Section: Introductionmentioning

confidence: 99%

“…L2-SP (Xuhong, Grandvalet, and Davoine 2018) uses an L2 penalty to restrict the updates of parameters, addressing knowledge forgetting during fine-tuning. However, it is not compatible with adaptive optimizers (Loshchilov and Hutter 2017;Guan 2023), which may produce the wrong regularization direction. Parameter isolation methods (Jia et al 2022;Sohn et al 2023) create new branches or modules for different network models and tasks for downstream tasks.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

One Step Learning, One Step Review

Huang,

Li,

et al. 2024

AAAI

View full text Add to dashboard Cite

Visual fine-tuning has garnered significant attention with the rise of pre-trained vision models. The current prevailing method, full fine-tuning, suffers from the issue of knowledge forgetting as it focuses solely on fitting the downstream training set. In this paper, we propose a novel weight rollback-based fine-tuning method called OLOR (One step Learning, One step Review). OLOR combines fine-tuning with optimizers, incorporating a weight rollback term into the weight update term at each step. This ensures consistency in the weight range of upstream and downstream models, effectively mitigating knowledge forgetting and enhancing fine-tuning performance. In addition, a layer-wise penalty is presented to employ penalty decay and the diversified decay rate to adjust the weight rollback levels of layers for adapting varying downstream tasks. Through extensive experiments on various tasks such as image classification, object detection, semantic segmentation, and instance segmentation, we demonstrate the general applicability and state-of-the-art performance of our proposed OLOR. Code is available at https://github.com/rainbow-xiao/OLOR-AAAI-2024.

show abstract

Section: Fine-tuning Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 96%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

One Step Learning, One Step Review

Huang,

Li,

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Other strategies have been proposed that use various metrics to choose more representative elements for the memory [10,16,17,2]. Some research has focused on the impact of hyperparameters on certain methods [33] or the effect of rehearsal methods on loss functions [46]. Other studies have explored methods for selecting elements from the memory, such as selecting elements based on how much their loss would be affected [1] or using a ranking based on the importance of preserving prior knowledge [20].…”

Section: Related Workmentioning

confidence: 99%

Memory Population in Continual Learning via Outlier Elimination

Hurtado,

Raymond-Sáez,

Araujo

et al. 2023

2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

Self Cite

View full text Add to dashboard Cite

Catastrophic forgetting, the phenomenon of forgetting previously learned tasks when learning a new one, is a major hurdle in developing continual learning algorithms. A popular method to alleviate forgetting is to use a memory buffer, which stores a subset of previously learned task examples for use during training on new tasks. The de facto method of filling memory is by randomly selecting previous examples. However, this process could introduce outliers or noisy samples that could hurt the generalization of the model. This paper introduces Memory Outlier Elimination (MOE), a method for identifying and eliminating outliers in the memory buffer by choosing samples from labelhomogeneous subpopulations. We show that a space with a high homogeneity is related to a feature space that is more representative of the class distribution. In practice, MOE removes a sample if it is surrounded by samples from different labels. We demonstrate the effectiveness of MOE on CIFAR-10, CIFAR-100, and CORe50, outperforming previous well-known memory population methods.

show abstract