Forward Compatible Few-Shot Class-Incremental Learning

Zhou, Dawei; Wang, Fuyun; Ye, Han-Jia; Ma, Liang; Pu, Shiliang; Zhan, De-Chuan

doi:10.1109/cvpr52688.2022.00884

Cited by 133 publications

(38 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Träuble et al (2021) proposed an efficient probabilistic approach to locate data instances whose old predictions could be incorrect and update them with ones from the new model. Zhou et al (2022) looked into forward compatibility, where new classes can be easily incorporated without negatively impacting existing prediction behavior. More recently, Schumann et al (2023) inspected classification model regression during training data updates and mitigated the problem by interpolating between weights of the old and new models.…”

Section: Discussionmentioning

confidence: 99%

Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion

Lai¹,

Mansimov²,

Xie³

et al. 2023

Preprint

View full text Add to dashboard Cite

When upgrading neural models to a newer version, new errors that were not encountered in the legacy version can be introduced, known as regression 1 errors. This inconsistent behavior during model upgrade often outweighs the benefits of accuracy gain and hinders the adoption of new models. To mitigate regression errors from model upgrade, distillation and ensemble have proven to be viable solutions without significant compromise in performance. Despite the progress, these approaches attained an incremental reduction in regression which is still far from achieving backwardcompatible model upgrade. In this work, we propose a novel method, Gated Fusion, that promotes backward compatibility via learning to mix predictions between old and new models. Empirical results on two distinct model upgrade scenarios show that our method reduces the number of regression errors by 62% on average, outperforming the strongest baseline by an average of 25%. * Work done during author's internship at AWS AI Labs. 1 Within this work, regression denotes performance degradation in software systems, instead of the statistical technique for estimating relationships among variables.

show abstract

Section: Discussionmentioning

confidence: 99%

Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion

Lai¹,

Mansimov²,

Xie³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Great efforts have been devoted to the following two directions: identifying and preserving significant parameters of the original model [32], and memorizing the knowledge of the old model through some strategies like knowledge distillation [31]. Recently, some works focused on generalizing CIL to a limited-data regime and led to a new practical scenario, i.e., Few-Shot CIL (FSCIL) [9], [10], [11], [12], [13], [14], [15], [16]. Existing methods for FSCIL mainly employ two strategies.…”

Section: B Class-incremental Learningmentioning

confidence: 99%

“…Based on a model well-trained by a large-scale base dataset, FSCIL [9], [10], [11], [12], [13], [14], [15], [16] aims to incrementally learn new classes in limited labeled data regime without forgetting previously seen categories. This emerging research topic faces the following challenges:…”

Section: Introductionmentioning

confidence: 99%

Uncertainty-Guided Semi-Supervised Few-Shot Class-Incremental Learning With Knowledge Distillation

Cui

Deng²,

Xu³

et al. 2023

IEEE Trans. Multimedia

View full text Add to dashboard Cite

Given a model well-trained with a large-scale base dataset, Few-Shot Class-Incremental Learning (FSCIL) aims at incrementally learning novel classes from a few labeled samples by avoiding overfitting, without catastrophically forgetting all encountered classes previously. Currently, semi-supervised learning technique that harnesses freely-available unlabeled data to compensate for limited labeled data can boost the performance in numerous vision tasks, which heuristically can be applied to tackle issues in FSCIL, i.e., the Semi-supervised FSCIL (Semi-FSCIL). So far, very limited work focuses on the Semi-FSCIL task, leaving the adaptability issue of semi-supervised learning to the FSCIL task unresolved. In this paper, we focus on this adaptability issue and present a simple yet efficient Semi-FSCIL framework named Uncertainty-aware Distillation with Class-Equilibrium (UaD-CE), encompassing two modules UaD and CE. Specifically, when incorporating unlabeled data into each incremental session, we introduce the CE module that employs a class-balanced self-training to avoid the gradual dominance of easy-to-classified classes on pseudo-label generation. To distill reliable knowledge from the reference model, we further implement the UaD module that combines uncertainty-guided knowledge refinement with adaptive distillation. Comprehensive experiments on three benchmark datasets demonstrate that our method can boost the adaptability of unlabeled data with the semi-supervised learning technique in FSCIL tasks. The code is available at https://github.com/yawencui/UaD-CE.

show abstract

“…We also introduce prompt regularization to improve performance and prevent forgetting. Our experimental results demonstrate that CPE-CLIP significantly improves FSCIL performance compared to state-of-the-art proposals while also drastically reducing the number of learnable parameters and training costs.Recent research has focused on solving these problems through various approaches, such as meta-learning [57, 34], regularization techniques [30], or knowledge distillation [38,6,62]. These methods have shown promising results in achieving incremental learning over time with a limited amount of data available.…”

mentioning

confidence: 99%

“…Recent research has focused on solving these problems through various approaches, such as meta-learning [57, 34], regularization techniques [30], or knowledge distillation [38,6,62]. These methods have shown promising results in achieving incremental learning over time with a limited amount of data available.…”

mentioning

confidence: 99%

Multimodal Parameter-Efficient Few-Shot Class Incremental Learning

D’Alessandro¹,

Alonso²,

Calabrés³

et al. 2023

Preprint

View full text Add to dashboard Cite

Few-Shot Class Incremental Learning (FSCIL) is a challenging continual learning task, where limited training examples are available during several learning sessions. To succeed in this task, it is necessary to avoid over-fitting new classes caused by biased distributions in the few-shot training sets. The general approach to address this issue involves enhancing the representational capability of a predefined backbone architecture by adding special modules for backward compatibility with older classes. However, this approach has not yet solved the dilemma of ensuring high classification accuracy over time while reducing the gap between the performance obtained on larger training sets and the smaller ones. In this work, we propose an alternative approach called Continual Parameter-Efficient CLIP (CPE-CLIP) to reduce the loss of information between different learning sessions. Instead of adapting additional modules to address information loss, we leverage the vast knowledge acquired by CLIP in large-scale pre-training and its effectiveness in generalizing to new concepts. Our approach is multimodal and parameter-efficient, relying on learnable prompts for both the language and vision encoders to enable transfer learning across sessions. We also introduce prompt regularization to improve performance and prevent forgetting. Our experimental results demonstrate that CPE-CLIP significantly improves FSCIL performance compared to state-of-the-art proposals while also drastically reducing the number of learnable parameters and training costs.Recent research has focused on solving these problems through various approaches, such as meta-learning [57, 34], regularization techniques [30], or knowledge distillation [38,6,62]. These methods have shown promising results in achieving incremental learning over time with a limited amount of data available. The general approaches consist in enhancing the basic representational capability of a predefined backbone architecture by adding special modules to entail backward compatibility with older classes during learning sessions. These solutions are computationally expensive since they need a large number of iterations in each session to adapt the additional modules to new classes while maintaining backward compatibility. Despite the high computational cost, they still fail to efficiently reduce the gap between the performance obtained on larger training sets

show abstract

Forward Compatible Few-Shot Class-Incremental Learning

Cited by 133 publications

References 33 publications

Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion

Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion

Uncertainty-Guided Semi-Supervised Few-Shot Class-Incremental Learning With Knowledge Distillation

Multimodal Parameter-Efficient Few-Shot Class Incremental Learning

Contact Info

Product

Resources

About