Recovery of Unrehearsed Items in Connectionist Models

Abstract

W hen gradient-descent models with hidden units are retrained on a portion of a previously lear ned set of items, performance on both the relear ned and unrelear ned items improves. Previous explanations of this phenom enon have not adequately distinguished recover y, which is dependent on original learning, from generalization, which is independent of original learning. Using a measure of vector similarity to track global changes in the weight state of three-layer networks, we show that (a) unlike in networks without hidden units, recover y occurs in the absence of generalization in networks with hidden units, and (b) when the conditions of lear ning are varied, changes in the extent of recover y are re¯ected in changes in the extent to which the weights move back towards their values held after original lear ning. The implications of this work for rehabilitation studies, hum an relear ning and models of hum an long-te rm memory are also considered.KEYW ORDS: Gradient-descent m odels, relearning, m emory, backpropagation.

What happens when we relearn part of what we previously knew? Predictions and constraints for models of long-term memory

“…2 Recovery in the VOCAB network (20 runs). When the VOCAB network relearns a portion of the originally learned set of training patterns, performance on both sets R and U improves (from Atkins & Murre, 1998). Pattern error was de®ned as the sum of squares of the dierence between the actual response and the target response across all output units and across all items in the set, divided by the number of patterns so as to eliminate the eects of set size (Rumelhart et al, 1986, p. 323) 1998).…”

“…This was the procedure used by Hinton and Sejnowski (1986) to test their Boltzmann machine network. With Hebbian learning, the forgetting produced by this method is approximately equivalent to training on other random patterns, the approach used by Atkins and Murre (1998) and Hinton and Plaut (1987).…”

“…Recovery occurs in these models because additional learning of related or unrelated items in multi-layer, gradient descent networks results in previously acquired weights being partially reinstated (Atkins & Murre, 1998;Hinton & Plaut, 1987). In brief, this process depends upon optimization of performance on the entire training set during original learning (see Atkins & Murre, 1998, for a detailed description).…”

“…Plaut (1992Plaut ( , 1996 demonstrated recovery in a network designed to model reading processes (Hinton & Shallice, 1991) that used the back-propagation through time learning algorithm. Atkins and Murre (1998) demonstrated recovery in a standard feed-forward, back-propagation network designed to model foreign-language vocabulary acquisition called the VO-CAB network (Fig. 2).…”

“…In brief, this process depends upon optimization of performance on the entire training set during original learning (see Atkins & Murre, 1998, for a detailed description). During relearning, the networks are more likely to return to this optimal solution for the entire training set than they are to move to some other solution that optimizes performance solely on the subset of items that are relearned.…”