How Do PDP Models Learn Quasiregularity?

Abstract

Parallel Distributed Processing (PDP) models have had a profound impact on the study of cognition. One domain in which they have been particularly influential is quasiregular learning, in which mastery requires both learning regularities that capture the majority of the structure in the input plus learning exceptions that violate the regularities. How PDP models learn quasiregularity is still not well understood. Small-and large-scale analyses of a feedforward, three-layer network were carried out to address two fundamental issues about network functioning: how the model can learn both regularities and exceptions without sacrificing generalizability; and the nature of the hidden representation that makes this learning possible. Results show that capacity-limited learning pressures the network to form componential representations, which ensures good generalizability. Small and highly local perturbations of this representational system allow exceptions to be learned while minimally disrupting generalizability. Theoretical and methodological implications of the findings are discussed. Keywords PDP model; quasiregularity; network analysis; hidden representationThe Parallel Distributed Processing (PDP) approach to studying cognition has had a significant impact in cognitive science, ranging from the fields of perception to reasoning (Thomas & McClelland, 2008). The premise of the approach is that many neuron-like units interacting through inhibition and excitation can provide insight into not only the end-state of learning, but also crucially the learning process itself. That is, PDP models can be used as tools to understand how mastery of an ability or skill is achieved.A fundamental problem in learning to which PDP models have been applied is learning quasiregularity; that is, learning certain regularities, but also learning violations of those regularities. Quasiregularity is ubiquitous in how humans structure the environment. It is required to form categories of objects, learn concepts, read words, and solve problems. To highlight one example, quasiregularity is especially prevalent in language, and one focus of research in language processing is to understand how humans learn regular items, which can be grouped together and thus described by rules (e.g., verbs that become past tense by Correspondence adding ed), and at the same time learn exceptions, which violate such rules and require a context-sensitive response (e.g., the past tense of find being found, not finded). Rumelhart and McClelland (1987) showed that a PDP model can learn both the regularities and the exceptions.Despite the ability of PDP models to learn quasiregularity, how they do so is poorly understood. In particular, little is known about the representations the model forms that enable it to distinguish regulars from exceptions. More specifically, what is the nature of the transformation between input and output? The complexity of PDP models has made it difficult to answer such questions, yet answers are vital. A model is more useful ...

Generalization from newly learned words reveals structural properties of the human reading system.

“…The learning mechanism in which the representations of exceptions are warped is responsible for the reduced ability to generalize. Warping is highly local to neighbors of the anchor (Kim et al, 2013), which is why generalization of the newly learned pronunciation is low and regularization itself is high.…”

“…Warping 5 To better understand how internal representations in the Plaut et al (1996) model made it both generalize appropriately and learn exceptions, Kim, Pitt, and Myung (2013) performed a series of simulations to flesh out how representations of regularities and exceptions were organized in that model. Intuitively, the characteristics of the representations of regular and exception pronunciations would seem to be at odds: learning an exception like pint should hinder the model's ability to generalize its knowledge of how int should be pronounced when encountering new words (e.g., kint, bint, gint).…”

“…The hidden units make it possible for the model to learn complex mappings between input and output that are necessary in a deep orthography like English, and warping occurs in the connections between layers. We used the implementation of the model developed by Kim et al (2013), which was shown to perform equivalently to the Plaut et al version. To ensure the stability of our results, the simulation results that we report are averages over 50 different instances of the model, each initialized using different random initial weights.…”

“…To seek converging evidence that warping underlies these large effects, we assessed whether the Plaut et al (1996) model can simulate the behavioral data (i.e., those reported in Figure 6). As mentioned in the Introduction, the notion of warping originated from a detailed analysis of this model (Kim et al, 2013). To the extent that it is a viable approximation of representation formation in readers, and in quasiregular domains more generally, the model should reproduce the qualitative patterns found in the data using the same anchor and probe stimuli.…”

“…The learning mechanism in which the representations of exceptions are warped is responsible for the reduced ability to generalize. Warping is highly local to neighbors of the anchor (Kim et al, 2013), which is why generalization of the newly learned pronunciation is low and regularization itself is high.Inevitably, there are some differences at a quantitative level either in terms of the initial notraining regularization rates or in the relative change in regularization rates after training. But, as noted previously, given the vast differences between the simulation and the human participants on many fronts (e.g., total vocabulary size, amount of consolidation of existing vocabulary vs.…”