REMIND Your Neural Network to Prevent Catastrophic Forgetting

Hayes, Tyler L.; Kafle, Kushal; Shrestha, Robik; Acharya, Manoj; Kanan, Christopher

doi:10.1007/978-3-030-58598-3_28

Cited by 162 publications

(98 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many replay algorithms for DNNs store past observations, and during replay internally convert observations into a suitable feature space using a previously learned transformation, such as a convolutional network (e.g., Mnih et al, 2015). But the benefits of directly storing internal representations for replay are increasingly acknowledged (Kapturowski et al, 2019;Iscen et al, 2020;Caccia et al, 2019;Hayes et al, 2019;van de Ven et al, 2020;Pellegrini et al, 2019;Hayes et al, 2021). Amongst others, storing internal representations is often more memory efficient (Iscen et al, 2020;Hayes et al, 2019), while observations can still be recreated from compressed internal representations if they are needed (van de Ven et al, 2020).…”

Section: Replay Can Reflect State Representationsmentioning

confidence: 99%

Replay in minds and machines

Wittkuhn

Chien

Hall-McMaster

2021

Neuroscience & Biobehavioral Reviews

View full text Add to dashboard Cite

Experience-related brain activity patterns reactivate during sleep, wakeful rest, and brief pauses from active behavior. In parallel, machine learning research has found that experience replay can lead to substantial performance improvements in artificial agents. Together, these lines of research suggest replay has a variety of computational benefits for decision-making and learning. Here, we provide an overview of putative computational functions of replay as suggested by machine learning and neuroscientific research. We show that replay can lead to faster learning, less forgetting, reorganization or augmentation of experiences, and support planning and generalization. In addition, we highlight the benefits of reactivating abstracted internal representations rather than veridical memories, and discuss how replay could provide a mechanism to build internal representations that improve learning and decision-making.

show abstract

Section: Replay Can Reflect State Representationsmentioning

confidence: 99%

Replay in minds and machines

Wittkuhn

Chien

Hall-McMaster

2021

Neuroscience & Biobehavioral Reviews

View full text Add to dashboard Cite

show abstract

“…In computer vision, most of continual learning strategies applied on large-scale datasets use rehearsal learning: a limited amount of the training data of old classes is kept during training [55]. This data is usually kept in raw form (e.g., pixels) [54,5,10] but can also be compressed [29,34], or trimmed [18] to reduce memory overhead; others store only a model to generate new samples of past classes [35,60,44]. In addition, most approaches aim at limiting the changes in the model when new classes are learned.…”

Section: Related Workmentioning

confidence: 99%

DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion

Douillard¹,

Ramé²,

Couairon³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep network architectures struggle to continually learn new tasks without forgetting the previous tasks. A recent trend indicates that dynamic architectures based on an expansion of the parameters can reduce catastrophic forgetting efficiently in continual learning. However, existing approaches often require a task identifier at test-time, need complex tuning to balance the growing number of parameters, and barely share any information across tasks. As a result, they struggle to scale to a large number of tasks without significant overhead. In this paper, we propose a transformer architecture based on a dedicated encoder/decoder framework. Critically, the encoder and decoder are shared among all tasks. Through a dynamic expansion of special tokens, we specialize each forward of our decoder network on a task distribution. Our strategy scales to a large number of tasks while having negligible memory and time overheads due to strict control of the parameters expansion. Moreover, this efficient strategy doesn't need any hyperparameter tuning to control the network's expansion. Our model reaches excellent results on CIFAR100 and state-of-the-art performances on the largescale ImageNet100 and ImageNet1000 while having less parameters than concurrent dynamic frameworks. Code will be released 1 .

show abstract

“…In addition, the system needs to store the original images as well as a GAN to generate images and such a process is memory demanding. The REMIND approach in a neural network that can be used to prevent catastrophic forgetting [27] is another Convolutional Neural Network (CNN)-based model that tackles the problem of catastrophic forgetting using rehearsal. Unlike the prior work where raw images of the previous classes are stored, these methods store quantized tensors for rehearsal, which is less memory demanding.…”

Section: Related Work and Motivationmentioning

confidence: 99%

A Novel Approach of IoT Stream Sampling and Model Update on the IoT Edge Device for Class Incremental Learning in an Edge-Cloud System

2021

View full text Add to dashboard Cite

With the exponential rise of the number of IoT devices, the amount of data being produced is massive. Thus, it is unfeasible to send all the raw data directly to the cloud for processing, especially for data that is high dimensional. Training deep learning models incrementally evolves the model over time and eliminates the need to statically training the models with all the data. However, the integration of class incremental learning and the Internet of Things (IoT) is a new concept and is not yet mature. In the context of IoT and deep learning, the transmission cost of data in the edge-cloud architecture is a challenge. We demonstrate a novel sample selection method that discards certain training images on the IoT edge device that reduces transmission cost and still maintains class incremental learning performance. It can be unfeasible to transmit all parameters of a trained model back to the IoT edge device. Therefore, we propose an algorithm to find only the useful parameters of a trained model in an efficient way to reduce the transmission cost from the cloud to the edge devices. Results show that our proposed methods can effectively perform class-incremental learning in an edge-cloud setting.

show abstract

REMIND Your Neural Network to Prevent Catastrophic Forgetting

Cited by 162 publications

References 58 publications

Replay in minds and machines

Replay in minds and machines

DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion

A Novel Approach of IoT Stream Sampling and Model Update on the IoT Edge Device for Class Incremental Learning in an Edge-Cloud System

Contact Info

Product

Resources

About