Sequence Recognition with Recurrent Neural Networks

Maskara, Arun; Noetzel, Andrew S.

doi:10.1080/09540099308915692

Cited by 17 publications

(11 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We will assume that the reader is familiar with the design of a Simple Recurrent Network (SRN, Elman, 1990). An RSRN ( Figure 2) works very much like a standard SRN (Maskara & Noetzel, 1992, 1993Cleeremans & Destrebecqz, 1997). Just as the RBP network involves adding "autoassociative" nodes to the output layer of a BP network, a reverberating SRN involves adding "autoassociative" nodes to the output layer of an SRN.…”

Section: Reverberating Simple Recurrent Network (Rsrn)mentioning

confidence: 99%

“…We will use a connectionist architecture using two coupled "auto-associative recurrent networks (AARN)" (Maskara & Noetzel, 1992, 1993Cleeremans & Destrebecqz, 1997) that pass information back and forth to each other by means of pseudopatterns. We will refer to auto-associative recurrent networks as Reverberating SRNs (RSRN), in order to emphasize the manner in which they use pseudopatterns to eliminate catastrophic interference in multiple sequence learning.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Preventing Catastrophic Interference in Multiple-Sequence Learning Using Coupled Reverberating Elman Networks

Ans¹,

Rousset²,

French³

et al. 2019

Proceedings of the Twenty-Fourth Annual Conference of the Cognitive Science Society

View full text Add to dashboard Cite

Everyone agree s that real cognition requires much more than static pattern recognition. In particular, it requires the ability to learn sequences of patterns (or actions) But learning sequences really means being able to learn multiple sequences, one after the other, wi thout the most recently learned ones erasing the previously learned ones. But if catastrophic interference is a problem for the sequential learning of individual patterns, the problem is amplified many times over when multiple sequences of patterns have to be learned consecutively, because each new sequence consists of many linked patterns. In this paper we will present a connectionist architecture that would seem to solve the problem of multiple sequence learning using pseudopatterns.

show abstract

Section: Reverberating Simple Recurrent Network (Rsrn)mentioning

confidence: 99%

mentioning

confidence: 99%

Preventing Catastrophic Interference in Multiple-Sequence Learning Using Coupled Reverberating Elman Networks

Ans¹,

Rousset²,

French³

et al. 2019

Proceedings of the Twenty-Fourth Annual Conference of the Cognitive Science Society

View full text Add to dashboard Cite

show abstract

“…In order to evaluate whether the failure of the model to account for human learning was particular to some aspect of the SRN's back‐propagation training procedure, we explored the possibility of making a number of changes to the training procedure. In particular, we considered changes which might improve the speed of learning in the model, such as changing the steepness of the sigmoid squashing function (Izui & Pentland, 1990), using a variant of back‐propagation called quick‐prop (Fahlman, 1988), changing the range of random values used to initialize the weights (from 0.001 to 2.0), and related variants of the SRN architecture such as the AARN (Maskara & Noetzel, 1992, 1993) in which the model predicts not only the next response but its current input and hidden unit activations. Under none of these circumstances were we able to find a pattern of learning which approximated human learning.…”

Section: Model‐based Analysesmentioning

confidence: 99%

Direct Associations or Internal Transformations? Exploring the Mechanisms Underlying Sequential Learning Behavior

Gureckis

Love

2010

Cognitive Science

View full text Add to dashboard Cite

We evaluate two broad classes of cognitive mechanisms that might support the learning of sequential patterns. According to the first, learning is based on the gradual accumulation of direct associations between events based on simple conditioning principles. The other view describes learning as the process of inducing the transformational structure that defines the material. Each of these learning mechanisms predicts differences in the rate of acquisition for differently organized sequences. Across a set of empirical studies, we compare the predictions of each class of model with the behavior of human subjects. We find that learning mechanisms based on transformations of an internal state, such as recurrent network architectures (e.g., Elman, 1990), have difficulty accounting for the pattern of human results relative to a simpler (but more limited) learning mechanism based on learning direct associations. Our results suggest new constraints on the cognitive mechanisms supporting sequential learning behavior.Keywords: Sequence learning; Skill acquisition and learning; Learning constraints; Simple recurrent networks; Associative learingThe ability to learn about the stream of events we experience allows us to perceive melody and rhythm in music, to coordinate the movement of our bodies, and to comprehend and produce utterances; it forms the basis of our ability to predict and anticipate. Despite the ubiquity of sequential learning in our mental lives, an understanding of the mechanisms which underlie this ability has proven elusive. Over the last hundred years, the field has offered at least two major perspectives concerning the mechanisms supporting this ability. The first (which we will call the ''Associationist'' or ''Behaviorist'' view) describes learning as the process of incrementally acquiring associative relationships between events (stimuli or responses) that are repeatedly paired. A defining feature of this paradigm is the claim that behavior can be characterized without reference to internal mental processes or representations (Skinner, 1957). For example, consider a student practicing typing at a keyboard by repeatedly entering the words ''we went camping by the water, and we got wet.'' According to associative chain theory (an influential associationist account of sequential processing), each action, such as pressing the letter c, is represented as a primitive node. Sequences of actions or stimuli are captured through unidirectional links that chain these nodes together (Ebbinghaus, 1964;Wickelgren, 1965). Through the process of learning, associations are strengthened between elements which often follow one another, so that, in this example, the link between the letters w and e is made stronger than the link between w and a because w-e is a more common subsequence. Through this process of incrementally adjusting weights between units that follow one another, the most activated unit at any point in time becomes the unit which should follow next in the sequence. Critically, the associationist pe...

show abstract

“…It is possible that the prediction task does not place sufficient demands on these networks to retain information from early time-points (since local information may be sufficient to predict subsequent segments). Alternatively, some more fundamental limitations on the memory capacity of recurrent neural networks for learning long distance dependencies provide the limiting factor on the performance of these systems (see, for instance, Servan-Schrieber, Cleeremans & McClelland, 1991;Maskara & Noetzel, 1993; see Rhode & Plaut, this volume for further discussion).…”

Section: Combining Multiple Cues For Segmentation and Identificationmentioning

confidence: 99%

Connectionist Modelling of Lexical Segmentation and Vocabulary Acquisition

Davis

2004

Connectionist Models of Development

View full text Add to dashboard Cite

Sequence Recognition with Recurrent Neural Networks

Cited by 17 publications

References 11 publications

Preventing Catastrophic Interference in Multiple-Sequence Learning Using Coupled Reverberating Elman Networks

Preventing Catastrophic Interference in Multiple-Sequence Learning Using Coupled Reverberating Elman Networks

Direct Associations or Internal Transformations? Exploring the Mechanisms Underlying Sequential Learning Behavior

Connectionist Modelling of Lexical Segmentation and Vocabulary Acquisition

Contact Info

Product

Resources

About