Convolutional Neural Network With Developmental Memory for Continual Learning

Park, Gyeong-Moon; Yoo, Sahng-Min; Kim, Jong-Hwan

doi:10.1109/tnnls.2020.3007548

Cited by 15 publications

(5 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…CL has already been successfully applied in a domain of interest for our study: image classification. For instance, [17] explore the usage of developmental memories for the damping of forgetting, and [18] apply CL by using transfer learning and k-nearest neighbor. While applications of CL for pattern recognition on accelerometer sensors are still pretty new, this field has been explored only in some standard TinyML applications [19].…”

Section: Related Workmentioning

confidence: 99%

Incremental Online Learning Algorithms Comparison for Gesture and Visual Smart Sensors

Avi

Albanese

Brunelli

2022

2022 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

Tiny machine learning (TinyML) in IoT systems exploits MCUs as edge devices for data processing. However, traditional TinyML methods can only perform inference, limited to static environments or classes. Real case scenarios usually work in dynamic environments, thus drifting the context where the original neural model is no more suitable. For this reason, pre-trained models reduce accuracy and reliability during their lifetime because the data recorded slowly becomes obsolete or new patterns appear. Continual learning strategies maintain the model up to date, with runtime fine-tuning of the parameters. This paper compares four state-of-the-art algorithms in two real applications: i) gesture recognition based on accelerometer data and ii) image classification. Our results confirm these systems' reliability and the feasibility of deploying them in tiny-memory MCUs, with a drop in the accuracy of a few percentage points with respect to the original models for unconstrained computing platforms.

show abstract

Section: Related Workmentioning

confidence: 99%

Incremental Online Learning Algorithms Comparison for Gesture and Visual Smart Sensors

Avi

Albanese

Brunelli

2022

2022 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

show abstract

“…Using a Bayesian NN, CBLN [17] preserves distinctive parameters for different datasets for retaining performance. Similarly, [4] introduced developmental memory (DM) into a CNN, continually growing submemory networks to preserve important features of learned tasks while allowing faster learning. Each sub-memory can store task-specific knowledge by using a memory loss function and preserve it during continual adaptations.…”

Section: Related Workmentioning

confidence: 99%

“…Methods to alleviate catastrophic forgetting can be classified into many categories, such as memory-based method [3], [4], [5], [6] that preserve activation of learned data, and penalization method that penalize change of learned knowledge through regularization [7] or modularization [8]. Among these methods, modularity-based parametric isolation method is particularly appealing, which is immune to catastrophic forgetting due to freezing parameters after a task has been learned Fig.…”

Section: Introductionmentioning

confidence: 99%

Revisiting Parameter Reuse to Overcome Catastrophic Forgetting in Neural Networks

Zhao¹,

Saxena²,

Cao³

2022

Preprint

View full text Add to dashboard Cite

Neural networks tend to forget previously learned knowledge when continuously learning on datasets with varying distributions, a phenomenon known as catastrophic forgetting. More significant distribution shifts among datasets lead to more forgetting. Recently, parameter-isolation-based approaches have shown great potential in overcoming forgetting with significant distribution shifts. However, they suffer from poor generalization as they fix the neural path for each dataset during training and require dataset labels during inference. In addition, they do not support backward knowledge transfer as they prioritize past data over future ones. In this paper, we propose a new adaptive learning method, named AdaptCL, that fully reuses and grows on learned parameters to overcome catastrophic forgetting and allows the positive backward transfer without requiring dataset labels. Our proposed technique adaptively grows on the same neural path by allowing optimal reuse of frozen parameters. Besides, it uses parameter-level data-driven pruning to assign equal priority to the data. We conduct extensive experiments on MNIST Variants and DomainNet datasets under different intensities of distribution shifts without requiring dataset labels. Results demonstrate that our proposed method is superior to alternative baselines in minimizing forgetting and enabling positive backward knowledge transfer.

show abstract

“…A rotated version of the training dataset is now assumed as the new batch of data that arrived over time. Figure (5) shows the training data in the top row and test data in the bottom row. Now, since the first CNN has a data augmentation module, the test data would not provide and specific challenge as the network has rotation invariant capabilities.…”

Section: Experiments For Dataset Distancementioning

confidence: 99%

Methods for maintenance of neural networks in continual learning scenarios

Suri¹,

Srivastava²,

Yeturu³

2021

Preprint

View full text Add to dashboard Cite

Neural networks suffer from catastrophic forgetting problem when deployed in a continual learning scenario where new batches of data arrive over time; however they are of different distributions from the previous data used for training the neural network. For assessing the performance of a model in a continual learning scenario, two aspects are important (i) to compute the difference in data distribution between a new and old batch of data and (ii) to understand the retention and learning behavior of deployed neural networks. Current techniques indicate the novelty of a new data batch by comparing its statistical properties with that of the old batch in the input space. However, it is still an open area of research to consider the perspective of a deployed neural network’s ability to generalize on the unseen data samples. In this work, we report a dataset distance measuring technique that indicates the novelty of a new batch of data while considering the deployed neural network’s perspective. We propose the construction of perspective histograms which are a vector representation of the data batches based on the correctness and confidence in the prediction of the deployed model. We have successfully tested the hypothesis empirically on image data coming MNIST Digits, MNIST Fashion, CIFAR10, for its ability to detect data perturbations of type rotation, Gaussian blur, and translation. Upon new data, given a model and its training data, we have proposed and evaluated four new scoring schemes, retention score (R), learning score (L), Oscore and SP-score for studying how much the model can retain its performance on past data, how much it can learn new data, the combined expression for the magnitude of retention and learning and stability-plasticity characteristics respectively. The scoring schemes have been evaluated MNIST Digits and MNIST Fashion data sets on different types of neural network architectures based on the number of parameters, activation functions, and learning loss functions, and an instance of a typical analysis report is presented. Machine learning model maintenance is a reality in production systems in the industry, and we hope our proposed methodology offers a solution to the need of the day in this aspect.

show abstract

Convolutional Neural Network With Developmental Memory for Continual Learning

Cited by 15 publications

References 20 publications

Incremental Online Learning Algorithms Comparison for Gesture and Visual Smart Sensors

Incremental Online Learning Algorithms Comparison for Gesture and Visual Smart Sensors

Revisiting Parameter Reuse to Overcome Catastrophic Forgetting in Neural Networks

Methods for maintenance of neural networks in continual learning scenarios

Contact Info

Product

Resources

About