On Learning the Geodesic Path for Incremental Learning

Simon, Christian; Koniusz, Piotr; Harandi, Mehrtash

doi:10.1109/cvpr46437.2021.00164

Cited by 88 publications

(50 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Aleatoric uncertainty and self-attention are involved in the proposed distillation loss. Simon et al [55] proposed to conduct knowledge distillation on the low-dimensional manifolds between the model outputs of previous and new tasks, which was shown to better mitigate catastrophic forgetting. Hu et al [56] found that the knowledge distillation based approaches do not have a consistent causal effect compared to end-to-end feature learning.…”

Section: Distillation In the Presence Of Sufficient Data Instancesmentioning

confidence: 99%

Recent Advances of Continual Learning in Computer Vision: An Overview

Qu,

Rahmani,

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Section: Distillation In the Presence Of Sufficient Data Instancesmentioning

confidence: 99%

Recent Advances of Continual Learning in Computer Vision: An Overview

Qu,

Rahmani,

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…[17] uses the causal effect on knowledge distillation to rectify class imbalance. [43] introduces geodesic path to traditional knowledge distillation. [1] combines task-wise knowledge distillation and separated softmax for bias compensation.…”

Section: Related Workmentioning

confidence: 99%

“…In the standard class-incremental learning [37,41,43], there are a sequence of streaming tasks T = {T t } T t=1 , where T denotes the task number, and the t-th task T t = {x t i , y t i } N t i=1 consists of N t pairs of samples x t i and their one-hot encoding labels y t i ∈ Y t . Y t represents the label space of the t-th task including C t new classes that are different from…”

Section: Problem Definitionmentioning

confidence: 99%

Federated Class-Incremental Learning

Dong¹,

Wang²,

Fang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Federated learning (FL) has attracted growing attentions via data-private collaborative training on decentralized clients. However, most existing methods unrealistically assume object classes of the overall framework are fixed over time. It makes the global model suffer from significant catastrophic forgetting on old classes in real-world scenarios, where local clients often collect new classes continuously and have very limited storage memory to store old classes. Moreover, new clients with unseen new classes may participate in the FL training, further aggravating the catastrophic forgetting of global model. To address these challenges, we develop a novel Global-Local Forgetting Compensation (GLFC) model, to learn a global classincremental model for alleviating the catastrophic forgetting from both local and global perspectives. Specifically, to address local forgetting caused by class imbalance at the local clients, we design a class-aware gradient compensation loss and a class-semantic relation distillation loss to balance the forgetting of old classes and distill consistent inter-class relations across tasks. To tackle the global forgetting brought by the non-i.i.d class imbalance across clients, we propose a proxy server that selects the best old global model to assist the local relation distillation. Moreover, a prototype gradient-based communication mechanism is developed to protect the privacy. Our model outperforms state-of-the-art methods by 4.4%∼15.1% in terms of average accuracy on representative benchmark datasets. The code is available at https://github. com/conditionWang/FCIL.

show abstract

“…Incremental learning: Incremental learning means learning from a sequence of data which appear over time. In the literature [33], incremental learning techniques are categorized into three groups, task-incremental learning [4,28,21], domain-incremental learning [39,29], and classincremental learning [26,1,13,36,31]. In this paper, we are only concerned with the third category, class-incremental learning, as we consider a unified output where the task label is not available during test time.…”

Section: Related Workmentioning

confidence: 99%

“…Wu et al [36] proposed a method for large scale incremental learning, where they correct the bias in the output of the model with the help of a linear model. Simon et al [31] propose a novel approach to the arsenal of distillation techniques. They construct lowdimensional manifolds for previous and current responses and minimize the dissimilarity between the geodesic responses connecting the manifolds.…”

Section: Related Workmentioning

confidence: 99%

Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning

Cheraghian¹,

Rahman²,

Fang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Few-shot class incremental learning (FSCIL) portrays the problem of learning new concepts gradually, where only a few examples per concept are available to the learner. Due to the limited number of examples for training, the techniques developed for standard incremental learning cannot be applied verbatim to FSCIL. In this work, we introduce a distillation algorithm to address the problem of FSCIL and propose to make use of semantic information during training. To this end, we make use of word embeddings as semantic information which is cheap to obtain and which facilitate the distillation process. Furthermore, we propose a method based on an attention mechanism on multiple parallel embeddings of visual data to align visual and semantic vectors, which reduces issues related to catastrophic forgetting. Via experiments on MiniImageNet, CUB200, and CI-FAR100 dataset, we establish new state-of-the-art results by outperforming existing approaches.

show abstract

On Learning the Geodesic Path for Incremental Learning

Cited by 88 publications

References 25 publications

Recent Advances of Continual Learning in Computer Vision: An Overview

Recent Advances of Continual Learning in Computer Vision: An Overview

Federated Class-Incremental Learning

Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning

Contact Info

Product

Resources

About