Continual lifelong learning with neural networks: A review

Parisi, German I.; Kemker, Ronald; Part, Jose L.; Kanan, Christopher; Wermter, Stefan

doi:10.1016/j.neunet.2019.01.012

Cited by 2,156 publications

(1,327 citation statements)

References 165 publications

Supporting

Mentioning

1,167

Contrasting

Unclassified

Order By: Relevance

“…As a result, the formation of cross-modal memories becomes a long-term dynamic process. Understanding the long-term dynamics of cortical memory representation in multimodal environment is not only a worthwhile topic by itself in brain research, but also significant for inspiring the enhancement of cross-model learning abilities of artificial brains [6].…”

Section: Introductionmentioning

confidence: 99%

Multimodal memory components and their long-term dynamics identified in cortical layers II/III but not layer Vb

Wang

Xie

et al. 2019

Preprint

View full text Add to dashboard Cite

Activity patterns of cerebral cortical regions represent the present environment in which animals receive multi-modal inputs. They are also shaped by the history of previous activity that reflects learned information on past multimodal exposures. We studied the long-term dynamics of cortical activity patterns during the formation of multimodal memories by analysing in vivo high-resolution 2-photon mouse brain imaging of Immediate Early Gene expression, resolved by cortical layers. Strikingly, in layers II/III, the patterns showed similar dynamics across functional distinct cortical areas and the consistency of dynamic patterns lasts for one to several days. In contrast, in layer Vb, the activity dynamics varied across functional distinct areas, and the present activities are sensitive to the previous activities at different time depending on the cortical locations, indicating that the information stored in the cortex at different time points is distributed across different cortical areas. These results suggest different roles of layer II/III and layer Vb neurons in the long-term multimodal perception of the environment.

show abstract

Section: Introductionmentioning

confidence: 99%

Multimodal memory components and their long-term dynamics identified in cortical layers II/III but not layer Vb

Wang

Xie

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…However, multitask models could help improve generalizability when introduced as an intermediary. Lifelong learning has been used to help models learn from separate but related tasks in stages or continuously and could be used to facilitate multitask training . Lifelong learning protocols have been used to transfer knowledge acquired on old tasks to new ones to improve generalization and facilitate model convergence.…”

Section: Introductionmentioning

confidence: 99%

A convolutional neural network algorithm for automatic segmentation of head and neck organs at risk using deep lifelong learning

et al. 2019

View full text Add to dashboard Cite

Purpose This study suggests a lifelong learning‐based convolutional neural network (LL‐CNN) algorithm as a superior alternative to single‐task learning approaches for automatic segmentation of head and neck (OARs) organs at risk. Methods and materials Lifelong learning‐based convolutional neural network was trained on twelve head and neck OARs simultaneously using a multitask learning framework. Once the weights of the shared network were established, the final multitask convolutional layer was replaced by a single‐task convolutional layer. The single‐task transfer learning network was trained on each OAR separately with early stoppage. The accuracy of LL‐CNN was assessed based on Dice score and root‐mean‐square error (RMSE) compared to manually delineated contours set as the gold standard. LL‐CNN was compared with 2D‐UNet, 3D‐UNet, a single‐task CNN (ST‐CNN), and a pure multitask CNN (MT‐CNN). Training, validation, and testing followed Kaggle competition rules, where 160 patients were used for training, 20 were used for internal validation, and 20 in a separate test set were used to report final prediction accuracies. Results On average contours generated with LL‐CNN had higher Dice coefficients and lower RMSE than 2D‐UNet, 3D‐Unet, ST‐ CNN, and MT‐CNN. LL‐CNN required ~72 hrs to train using a distributed learning framework on 2 Nvidia 1080Ti graphics processing units. LL‐CNN required 20 s to predict all 12 OARs, which was approximately as fast as the fastest alternative methods with the exception of MT‐CNN. Conclusions This study demonstrated that for head and neck organs at risk, LL‐CNN achieves a prediction accuracy superior to all alternative algorithms.

show abstract

“…As for neural network systems, the most straightforward and pragmatic method to avoid catastrophic forgetting is to retrain a deep learning model completely from scratch with all the old data and new data (Parisi et al, 2018). However, this method is proved to be very inefficient (Parisi et al, 2018). Moreover, the new model learned from scratch may share very low similarity with the old one, which results in poor learning robustness.…”

Section: Introductionmentioning

confidence: 99%

“…To accommodate the new knowledge, these methods dynamically allocate neural resources or retrain the model with an increasing number of neurons or layers. Intuitively, these approaches can prevent catastrophic forgetting but may also lead to scalability and generalization issues due to the increasing complexity of the network (Parisi et al, 2018). The last category utilizes the dual-memory learning system, which is inspired by the CLS theory (Hinton and Plaut, 1987;Lopez-Paz and Ranzato, 2017;Gepperth and Karaoguz, 2016).…”

Section: Introductionmentioning

confidence: 99%

SupportNet: a novel incremental learning framework through deep learning and support data

Liu

Ding

et al. 2018

Preprint

View full text Add to dashboard Cite

Motivation: In most biological data sets, the amount of data is regularly growing and the number of classes is continuously increasing. To deal with the new data from the new classes, one approach is to train a classification model, e.g., a deep learning model, from scratch based on both old and new data. This approach is highly computationally costly and the extracted features are likely very different from the ones extracted by the model trained on the old data alone, which leads to poor model robustness. Another approach is to fine tune the trained model from the old data on the new data. However, this approach often does not have the ability to learn new knowledge without forgetting the previously learned knowledge, which is known as the catastrophic forgetting problem. To our knowledge, this problem has not been studied in the field of bioinformatics despite its existence in many bioinformatic problems. Results: Here we propose a novel method, SupportNet, to solve the catastrophic forgetting problem efficiently and effectively. SupportNet combines the strength of deep learning and support vector machine (SVM), where SVM is used to identify the support data from the old data, which are fed to the deep learning model together with the new data for further training so that the model can review the essential information of the old data when learning the new information. Two powerful consolidation regularizers are applied to ensure the robustness of the learned model. Comprehensive experiments on various tasks, including enzyme function prediction, subcellular structure classification and breast tumor classification, show that SupportNet drastically outperforms the state-of-the-art incremental learning methods and reaches similar performance as the deep learning model trained from scratch on both old and new data.

show abstract

Continual lifelong learning with neural networks: A review

Cited by 2,156 publications

References 165 publications

Multimodal memory components and their long-term dynamics identified in cortical layers II/III but not layer Vb

Multimodal memory components and their long-term dynamics identified in cortical layers II/III but not layer Vb

A convolutional neural network algorithm for automatic segmentation of head and neck organs at risk using deep lifelong learning

SupportNet: a novel incremental learning framework through deep learning and support data

Contact Info

Product

Resources

About