Transfer Without Forgetting

Boschini, Matteo; Bonicelli, Lorenzo; Porrello, Angelo; Bellitto, Giovanni; Pennisi, Matteo; Palazzo, Simone; Spampinato, Concetto; Calderara, Simone

doi:10.1007/978-3-031-20050-2_40

Cited by 22 publications

(10 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To assess our proposal, we perform a suite of experiments through the Mammoth framework [12,17,9,6,27,13,14,50], an open-source codebase introduced in [15] for testing CL algorithms. In particular, we show that our method can be easily applied to state-of-the-art replay methods and enhance their performance in a wide variety of challenging settings and backbone architectures.…”

Section: Methodsmentioning

confidence: 99%

On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning

Bonicelli¹,

Boschini²,

Porrello³

et al. 2022

Preprint

View full text Add to dashboard Cite

Rehearsal approaches enjoy immense popularity with Continual Learning (CL) practitioners. These methods collect samples from previously encountered data distributions in a small memory buffer; subsequently, they repeatedly optimize on the latter to prevent catastrophic forgetting. This work draws attention to a hidden pitfall of this widespread practice: repeated optimization on a small pool of data inevitably leads to tight and unstable decision boundaries, which are a major hindrance to generalization. To address this issue, we propose Lipschitz-DrivEn Rehearsal (LiDER), a surrogate objective that induces smoothness in the backbone network by constraining its layer-wise Lipschitz constants w.r.t. replay examples. By means of extensive experiments, we show that applying LiDER delivers a stable performance gain to several state-of-the-art rehearsal CL methods across multiple datasets, both in the presence and absence of pre-training. Through additional ablative experiments, we highlight peculiar aspects of buffer overfitting in CL and better characterize the effect produced by LiDER. Code is available at https://github.com/aimagelab/LiDER.

show abstract

Section: Methodsmentioning

confidence: 99%

On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning

Bonicelli¹,

Boschini²,

Porrello³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In this section, we provide comprehensive per-step comparative results of our Baseline and Baseline++, juxtaposed with the adapted state-of-the-art methods on two task splits (two-step and five-step) of CIFAR-10 (C10) [28], CIFAR-100 (C100) [28], TinyImageNet-200 (T200) [29], 3 https://github.com/aimagelab/mammoth CUB-200 (B200) [42] and Herbarium-683 (H683) [38] As depicted in the reported figures, the overall accuracy exhibits a decline as the task sequence progresses, whereas the maximum forgetting for the novel classes discovered during the first step experiences an increase, attributable to the catastrophic forgetting issue [41]. In the context of longer task sequences (five-step split, as observed in the top half of the figures), the forgetting issue is exacerbated due to more frequent model updates.…”

Section: Detailed Experimental Resultsmentioning

confidence: 99%

“…All adapted methods unfreeze only the last transformer block of the feature extractor, except ResTune that unfreezes the last two blocks. This is because it is unnecessary to unlock all blocks of the largescale pre-trained model, as previously observed in [45] and [3]. Additional implementation details of the adapted methods are provided in the supplementary material.…”

Section: Comparison With the State-of-the-art Methodsmentioning

confidence: 99%

“…B.1. Prior to forward-Splits Two-step Five-step Task T [1] T [2] T [1] T [2] T [3] T [4] T [5] Datasets C [t] and the number of unlabelled images |D [t] | in D [t] for each task T [t] are reported.…”

Section: B2 Simple Yet Strong Baselines For Msc-incdmentioning

confidence: 99%

“…Thus, our result sets a precedent to future MSc-iNCD methods, while urging them to meticulously compare with baselines, that are as simple as having a frozen backbone and a linear classifier (see Sec. 3…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A cache cooperation management for wireless

Xiang

Zhong²,

Zhong³

2001 International Conferences on Info-Tech and Info-Net. Proceedings (Cat. No.01EX479)

View full text Add to dashboard Cite

Discovering novel concepts from unlabelled data and in a continuous manner is an important desideratum of lifelong learners. In the literature such problems have been partially addressed under very restricted settings, where either access to labelled data is provided for discovering novel concepts (e.g., NCD) or learning occurs for a limited number of incremental steps (e.g., class-iNCD). In this work we challenge the status quo and propose a more challenging and practical learning paradigm called MSc-iNCD, where learning occurs continuously and unsupervisedly, while exploiting the rich priors from large-scale pre-trained models. To this end, we propose simple baselines that are not only resilient under longer learning scenarios, but are surprisingly strong when compared with sophisticated state-of-theart methods. We conduct extensive empirical evaluation on a multitude of benchmarks and show the effectiveness of our proposed baselines, which significantly raise the bar.

show abstract