Catastrophic Forgetting in Connectionist Networks

French, Robert M.

doi:10.1002/0470018860.s00096

Cited by 116 publications

(144 citation statements)

References 12 publications

Supporting

Mentioning

144

Contrasting

Order By: Relevance

“…The rapid learning that we have observed is not typically associated with networks trained using back propagation, which often exhibit a trade-off between the speed of new learning and the stability of previously acquired knowledge (French, 1999;McClelland, McNaughton, & O'Reilly, 1995;Page, 2000). However, other supervised learning algorithms exist, including those in which sparse or localist representations mediate between the speech input and lexical output, and these might be capable of simulating a more rapid learning process.…”

Section: Implications For Models Of Speech Perceptionmentioning

confidence: 91%

Lexical Information Drives Perceptual Learning of Distorted Speech: Evidence From the Comprehension of Noise-Vocoded Sentences.

Davis¹,

Johnsrude²,

Hervais-Adelman³

et al. 2005

Journal of Experimental Psychology: General

463

583

View full text Add to dashboard Cite

Speech comprehension is resistant to acoustic distortion in the input, reflecting listeners' ability to adjust perceptual processes to match the speech input. For noise-vocoded sentences, a manipulation that removes spectral detail from speech, listeners' reporting improved from near 0% to 70% correct over 30 sentences (Experiment 1). Learning was enhanced if listeners heard distorted sentences while they knew the identity of the undistorted target (Experiments 2 and 3). Learning was absent when listeners were trained with nonword sentences (Experiments 4 and 5), although the meaning of the training sentences did not affect learning (Experiment 5). Perceptual learning of noise-vocoded speech depends on higher level information, consistent with top-down, lexically driven learning. Similar processes may facilitate comprehension of speech in an unfamiliar accent or following cochlear implantation.

show abstract

Section: Implications For Models Of Speech Perceptionmentioning

confidence: 91%

Lexical Information Drives Perceptual Learning of Distorted Speech: Evidence From the Comprehension of Noise-Vocoded Sentences.

Davis¹,

Johnsrude²,

Hervais-Adelman³

et al. 2005

Journal of Experimental Psychology: General

463

583

View full text Add to dashboard Cite

show abstract

“…This is in contrast to the behavior of, e.g., multilayer perceptrons [17], where retraining with slightly different input statistics can lead to a complete reorganization of the hidden layer structure, and therefore to a loss of already learned capabilities.…”

Section: Avoidance Of Catastrophic Forgettingmentioning

confidence: 75%

“…While this will certainly generate an induced representation, this solution is unsuitable because the hidden layer, while manifestly two-dimensional irrespectively of input data, would lack any topological organization if the MLP is trained using a back-propagation learning algorithm [15]. Furthermore, issues of catastrophic forgetting [17] would complicate the use of MLP still further. As an alternative, PCA produces, for each input, a set coordinates in the space of principal components which are not in any way topologically organized.…”

Section: Critical Examination and Justification Of Used Methodsmentioning

confidence: 99%

Efficient online bootstrapping of sensory representations

Gepperth

2013

Neural Networks

View full text Add to dashboard Cite

This is a simulation-based contribution exploring a novel approach to the open-ended formation of multimodal representations in autonomous agents. In particular, we address the issue of transferring ("bootstrapping") feature selectivities between two modalities, from a previously learned or innate reference representation to a new induced representation. We demonstrate the potential of this algorithm by several experiments with synthetic inputs modeled after a robotics scenario where multimodal object representations are "bootstrapped" from a (reference) representation of object affordances. We focus on typical challenges in autonomous agents: absence of human supervision, changing environment statistics and limited computing power. We propose an autonomous and local neural learning algorithm termed PRO-PRE (projection-prediction) that updates induced representations based on predictability: competitive advantages are given to those feature-sensitive elements that are inferable from activities in the reference representation. PRO-PRE implements a bi-directional interaction of clustering ("projection") and inference ("prediction"), the key ingredient being an efficient online measure of predictability controlling learning in the projection step. We show that the proposed method is computationally efficient and stable, and that the multimodal transfer of feature selectivity is successful and robust under resource constraints. Furthermore, we successfully demonstrate robustness to noisy reference representations, non-stationary input statistics and uninformative inputs. to be an open-ended and largely unsupervised process, leading to internal representations of ever increasing specialization and usefulness to the agent. The labeled boxes indicate the work covered in this article: I 1 , I 2 , I 3 stand for input representations derived from sensors, N indicates the newly formed induced representation, and R stands for the reference representation that is derived from another sensory modality, controlling the bootstrapping process.

show abstract

“…For the shape categories the SLP network architecture is only superior at earlier learning epochs, but is worse if the learning process is continued. Overall the SLP performance is surprisingly good, which is in contrast to classification tasks with a one-out-of-n class selection, where the SLP approach is known for the "catastrophic forgetting effect" (French, 1999). For our categorization task this effect is only slightly visible for the shape categories, where the performance increase for newly presented objects is distinctly less than for all other tested approaches.…”

Section: Color and Parts-based Featuresmentioning

confidence: 89%

“…Therefore in this paper we are particularly interested in incremental learning of representations under the condition, where a particular training vector can only be accessed for a limited time period. As a consequence the training with such a changing data ensemble typically causes the well-known "catastrophic forgetting effect" (French, 1999): With the incorporation of newly acquired knowledge, the previously learned knowledge is quickly fading out. Closely related to this effect is the term "catastrophic interference" (McCloskey & Cohen, 1989): Patterns of different categories which are similar in feature space, confuse the learning and overwrite earlier presented patterns.…”

Section: Introductionmentioning

confidence: 99%

A life-long learning vector quantization approach for interactive learning of multiple categories

et al. 2012

View full text Add to dashboard Cite

We present a new method capable of learning multiple categories in an interactive and lifelong learning fashion to approach the "stability-plasticity dilemma". The problem of incremental learning of multiple categories is still largely unsolved. This is especially true for the domain of cognitive robotics, requiring real-time and interactive learning. To achieve the life-long learning ability for a cognitive system, we propose a new learning vector quantization approach combined with a category-specific feature selection method to allow several metrical "views" on the representation space of each individual vector quantization node. These category-specific features are incrementally collected during the learning process, so that a balance between the correction of wrong representations and the stability of acquired knowledge is achieved. We demonstrate our approach for a difficult visual categorization task, where the learning is applied for several complex-shaped objects rotated in depth.

show abstract

Catastrophic Forgetting in Connectionist Networks

Abstract: Unlike human brains, connectionist networks can forget previously learned information suddenly and completely (‘catastrophically’) when learning new information. Various solutions to this problem have been proposed.

Cited by 116 publications

References 12 publications

Lexical Information Drives Perceptual Learning of Distorted Speech: Evidence From the Comprehension of Noise-Vocoded Sentences.

Lexical Information Drives Perceptual Learning of Distorted Speech: Evidence From the Comprehension of Noise-Vocoded Sentences.

Efficient online bootstrapping of sensory representations

A life-long learning vector quantization approach for interactive learning of multiple categories

Contact Info

Product

Resources

About