Avoiding catastrophic forgetting by coupling two reverberating neural networks

Ans, Bernard; Rousset, Stéphane

doi:10.1016/s0764-4469(97)82472-9

Cited by 73 publications

(86 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pseudorehearsal uses patterns generated from the network itself (known as pseudopatterns) during the rehearsal process (e.g. [19], [1], [9]). Random data is fed into the network, and the resulting output is stored.…”

Section: A Alleviating Catastrophic Interferencementioning

confidence: 99%

Mitigation of catastrophic forgetting in recurrent neural networks using a Fixed Expansion Layer

Coop

Arel

2013

The 2013 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

Abstract-Catastrophic forgetting (or catastrophic interference) in supervised learning systems is the drastic loss of previously stored information caused by the learning of new information. While substantial work has been published on addressing catastrophic forgetting in memoryless supervised learning systems (e.g. feedforward neural networks), the problem has received limited attention in the context of dynamic systems, particularly recurrent neural networks. In this paper, we introduce a solution for mitigating catastrophic forgetting in RNNs based on enhancing the Fixed Expansion Layer (FEL) neural network which exploits sparse coding of hidden neuron activations. Simulation results on several non-stationary data sets clearly demonstrate the effectiveness of the proposed architecture.

show abstract

Section: A Alleviating Catastrophic Interferencementioning

confidence: 99%

Mitigation of catastrophic forgetting in recurrent neural networks using a Fixed Expansion Layer

Coop

Arel

2013

The 2013 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

show abstract

“…[9] French, 1997;Ans and Rousset, 1997;French, Ans, & Rousset, 2001;Ans et al, 2002. [10] This is an improvement on previous work in which catastrophic forgetting was avoided for networks that trained on 'a series of, ''static'' (non-temporal) patterns' but not for temporal patterns (Ans & Rousset, 1997;French, 1997;Robbins, 1995 Ans et al, 2002.…”

Section: Notesmentioning

confidence: 99%

“…[10] This is an improvement on previous work in which catastrophic forgetting was avoided for networks that trained on 'a series of, ''static'' (non-temporal) patterns' but not for temporal patterns (Ans & Rousset, 1997;French, 1997;Robbins, 1995 Ans et al, 2002. [14] Incidentally, the RSRN approach of using pseudo-patterns seems to solve both the creation and storage of such representative knowledge.…”

Section: Notesmentioning

confidence: 99%

Atomistic learning in non-modular systems

Blackmon¹,

Byrd²,

Cummins³

et al. 2005

Philosophical Psychology

View full text Add to dashboard Cite

We argue that atomistic learning-learning that requires training only on a novel item to be learned-is problematic for networks in which every weight is available for change in every learning situation. This is potentially significant because atomistic learning appears to be commonplace in humans and most non-human animals. We briefly review various proposed fixes, concluding that the most promising strategy to date involves training on pseudo-patterns along with novel items, a form of learning that is not strictly atomistic, but which looks very much like it 'from the outside'. The ProblemThe assumption that we have the ability to evaluate and add or delete beliefs individually is common in the psychological literature on memory, concept acquisition, and language acquisition (Ramsey, Stitch, & Garon, 1990;Cummins, Poirier, & Roth, 2004). Indeed, this supposition pervades our informal, commonsense framework for understanding the mind, as well as our formal-symbolic models of rationality and epistemology. Rationality, for example, is thought of as at least in part about the management of beliefs and other propositional attitudes: what beliefs we should and should not adopt, when and what to add, when and what to delete, etc. Models of rationality tell us when we ought to revise our individual beliefs, and because ought implies can, these models presuppose that we can manage our beliefs individually. 1At the same time, most of our current connectionist models of cognition suggest that the knowledge used in carrying out many cognitive tasks related to memory,

show abstract

“…In this work, we assume that the inputs of the base patterns are available for similarity checking. The similarity checking here is functionally similar to the reverberating neural networks suggested in [7], where the random inputs are fed into an auto-associative sub-network so that the random inputs will converge to the most similar base pattern input.…”

Section: Simulation Studiesmentioning

confidence: 99%

“…30, 63073 Offenbach, Germany (email: yaochu.jin@honda-ri. de) A further step to go from the semi-distributed representations is to adopt a dual-network structure [15], [7] or a complementary learning systems [9], which belong to the third category. In these methods, one sub-structure is responsible for learning new patterns, and the other for consolidating the previously learned patterns.…”

Section: Introductionmentioning

confidence: 99%

Alleviating Catastrophic Forgetting via Multi-Objective Learning

Jin¹,

Sendhoff²

The 2006 IEEE International Joint Conference on Neural Network Proceedings

View full text Add to dashboard Cite

Abstract-Handling catastrophic forgetting is an interesting and challenging topic in modeling the memory mechanisms of the human brain using machine learning models. From a more general point of view, catastrophic forgetting reflects the stability-plasticity dilemma, which is one of the several dilemmas to be addressed in learning systems: to retain the stored memory while learning new information. Different to the existing approaches, we introduce a Pareto-optimality based multi-objective learning framework for alleviating catastrophic learning. Compared to the single-objective learning methods, multi-objective evolutionary learning with the help of pseudorehearsal is shown to be more promising in dealing with the stability-plasticity dilemma.

show abstract

Avoiding catastrophic forgetting by coupling two reverberating neural networks

Cited by 73 publications

References 11 publications

Mitigation of catastrophic forgetting in recurrent neural networks using a Fixed Expansion Layer

Mitigation of catastrophic forgetting in recurrent neural networks using a Fixed Expansion Layer

Atomistic learning in non-modular systems

Alleviating Catastrophic Forgetting via Multi-Objective Learning

Contact Info

Product

Resources

About