SpaceNet: Make Free Space for Continual Learning

Sokar, Ghada; Mocanu, Decebal Constantin; Pechenizkiy, Mykola

doi:10.1016/j.neucom.2021.01.078

Cited by 42 publications

(30 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Xu and Zhu [50] propose reinforcement learning for selecting the optimal architecture to learn each new task. The work in [45] trains sparse deep neural networks and compresses the sparse connections of each task in the network. The work in [48] expands the network for new tasks while preserving previous task knowledge and incorporating the topology and attribute of network nodes.…”

Section: Related Workmentioning

confidence: 99%

DILF-EN framework for Class-Incremental Learning

Asad¹,

Joshi²,

Mazumder³

et al. 2021

Preprint

View full text Add to dashboard Cite

A class-incremental learning problem is characterized by training data becoming available in a phase-by-phase manner. Deep learning models suffer from catastrophic forgetting of the classes in the older phases as they get trained on the classes introduced in the new phase. In this work, we show that the effect of catastrophic forgetting on the model prediction varies with the change in orientation of the same image, which is a novel finding. Based on this, we propose a novel data-ensemble approach that combines the predictions for the different orientations of the image to help the model retain further information regarding the previously seen classes and thereby reduce the effect of forgetting on the model predictions. However, we cannot directly use the data-ensemble approach if the model is trained using traditional techniques. Therefore, we also propose a novel dual-incremental learning framework that involves jointly training the network with two incremental learning objectives, i.e., the class-incremental learning objective and our proposed data-incremental learning objective. In the dualincremental learning framework, each image belongs to two classes, i.e., the image class (for class-incremental learning) and the orientation class (for data-incremental learning). In class-incremental learning, each new phase introduces a new set of classes, and the model cannot access the complete training data from the older phases. In our proposed data-incremental learning, the orientation classes remain the same across all the phases, and the data introduced by the new phase in class-incremental learning acts as new training data for these orientation classes. We empirically demonstrate that the dual-incremental learning framework is vital to the data-ensemble approach. We apply our proposed approach to state-of-the-art class-incremental learning methods and empirically show that our framework significantly improves the performance of these methods. Our proposed method significantly improves the performance of the stateof-the-art method (AANets) on the CIFAR-100 dataset by absolute margins of 3.30%, 4.28%, 3.55%, 4.03%, for the number of phases P=50, 25, 10, and 5, respectively, which establishes the efficacy of the proposed work.

show abstract

Section: Related Workmentioning

confidence: 99%

DILF-EN framework for Class-Incremental Learning

Asad¹,

Joshi²,

Mazumder³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Then, few unimportant parameters are used to learn each task. SpaceNet (Sokar et al, 2021c) learns sparse sub-network for each task from scratch using dynamic sparse training (Mocanu et al, 2018;Hoefler et al, 2021) where the weights and the sparse topology are optimized simultaneously.…”

Section: Related Workmentioning

confidence: 99%

“…In this paper, we analyze the recent task-specific components method, SpaceNet (Sokar et al, 2021c). The motivations for choosing SpaceNet are: (1) unlike most task-specific component methods, it does not rely on task identity; making it applicable for class-IL.…”

Section: Spacenetmentioning

confidence: 99%

“…Architectures. We followed the architecture used by (Zenke et al, 2017;Sokar et al, 2021c) Baselines. We compare with the following five baselines: (1) Scratch: learning a single task from scratch using a sparse neural network with the same sparsity level used for one task in the task-specific components baselines, (2) SpaceNet, Section 3.2, (Sokar et al, 2021a), (3) SI, Section 3.3, (Zenke et al, 2017), (4) Random reuse: a modified version of KAN in which the neurons used for allocating new connections are subset of the free neurons (i.e.…”

Section: Datasets and Baselinesmentioning

confidence: 99%

“…In addition, the model is biased towards new classes due to the unavailability of old data . Few setups were proposed in the literature to alleviate these limitations: (1) Cosine normalization (Hou et al, 2019) which enforces balanced magnitudes of the output weights across all classes, (2) Multi-headed layer (Mittal et al, 2021) where each task has its own output layer, (3) Pre-defined layer (Sokar et al, 2021c) where a large number of output neurons is defined at t = 0.…”

Section: Softmax Classifier For Class-incremental Learningmentioning

confidence: 99%

See 2 more Smart Citations

Avoiding Forgetting and Allowing Forward Transfer in Continual Learning via Sparse Networks

Sokar¹,

Mocanu²,

Pechenizkiy³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Continual learning agents should incrementally learn a sequence of tasks while satisfying two main desiderata: accumulating on previous knowledge without forgetting and transferring previous relevant knowledge to help in future learning. Existing research largely focuses on alleviating the catastrophic forgetting problem. There, an agent is altered to prevent forgetting based solely on previous tasks. This hinders the balance between preventing forgetting and maximizing the forward transfer. In response to this, we investigate the stability-plasticity dilemma to determine which model components are eligible to be reused, added, fixed, or updated to achieve this balance. We address the class incremental learning scenario where the agent is prone to ambiguities between old and new classes. With our proposed Knowledge-Aware contiNual learner (KAN), we demonstrate that considering the semantic similarity between old and new classes helps in achieving this balance. We show that being aware of existing knowledge helps in: (1) increasing the forward transfer from similar knowledge, (2) reducing the required capacity by leveraging existing knowledge, (3) protecting dissimilar knowledge, and (4) increasing robustness to the class order in the sequence. We evaluated sequences of similar tasks, dissimilar tasks, and a mix of both constructed from the two commonly used benchmarks for class-incremental learning; CIFAR-10 and CIFAR-100.

show abstract

Theoretical Understanding of the Information Flow on Continual Learning Performance

Andle

Sekeh

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Continual Learning (CL) has generated attention as a method of avoiding Catastrophic Forgetting (CF) in the sequential training of neural networks, improving network efficiency and adaptability to different tasks. Additionally, CL serves as an ideal setting for studying network behavior and Forward Knowledge Transfer (FKT) between tasks. Pruning methods for CL train subnetworks to handle the sequential tasks which allows us to take a structured approach to investigating FKT. Sharing prior subnetworks' weights leverages past knowledge for the current task through FKT. Understanding which weights to share is important as sharing all weights can yield sub-optimal accuracy. This paper investigates how different sharing decisions affect the FKT between tasks. Through this lens we demonstrate how task complexity and similarity influence the optimal weight sharing decisions, giving insights into the relationships between tasks and helping inform decision making in similar CL methods. We implement three sequential datasets designed to emphasize variation in task complexity and similarity, reporting results for both ResNet-18 and VGG-16. By sharing in accordance with the decisions supported by our findings, we show that we can improve task accuracy compared to other sharing decisions.

show abstract

SpaceNet: Make Free Space for Continual Learning

Cited by 42 publications

References 10 publications

DILF-EN framework for Class-Incremental Learning

DILF-EN framework for Class-Incremental Learning

Avoiding Forgetting and Allowing Forward Transfer in Continual Learning via Sparse Networks

Theoretical Understanding of the Information Flow on Continual Learning Performance

Contact Info

Product

Resources

About