A Field Guide to Dynamical Recurrent Networks

Kremer, Stefan C.; Kolen, John F.

doi:10.1109/9780470544037

Cited by 82 publications

(63 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For this purpose, the parametric bias (PB) vector is learned simultaneously and unsupervised during normal training of the network. The prediction error with respect to the desired output is determined and backpropagated through time using the BPTT algorithm [9]. However, the error is not only used to correct all the synaptic weights present in the Elman-type network.…”

Section: Storagementioning

confidence: 99%

See 1 more Smart Citation

Action-Driven Perception for a Humanoid

Kleesiek

Badde

Wermter

et al. 2013

Communications in Computer and Information Science

View full text Add to dashboard Cite

Abstract. We present active object categorization experiments with a real humanoid robot. For this purpose, the training algorithm of a recurrent neural network with parametric bias has been extended with adaptive learning rates. This modification leads to an increase in training speed. Using this new training algorithm we conducted three experiments aiming at object categorization. While holding different objects in its hand, the robot executes a motor sequence that induces multi-modal sensory changes. During learning, these high-dimensional perceptions are 'engraved' in the network. Simultaneously, low-dimensional PB values emerge unsupervised. The geometrical relation of these PB vectors can then be exploited to infer relations between the original high dimensional time series characterizing different objects. Even sensations belonging to unknown objects can be discriminated from known (learned) ones and kept apart from each other reliably. Additionally, we show that the network tolerates noisy sensory signals very well.

show abstract

Section: Storagementioning

confidence: 99%

“…Mostly, the architecture is utilized to model the mirror neuron system [7,8]. Here we apply the variant proposed by Cuijpers et al [8] using an Elman-type structure [9] at its core. Furthermore, we modify the training algorithm to include adaptive learning rates for training of the weights, as well as the PB values.…”

Section: Theorymentioning

confidence: 99%

Action-Driven Perception for a Humanoid

Kleesiek

Badde

Wermter

et al. 2013

Communications in Computer and Information Science

View full text Add to dashboard Cite

show abstract

“…[10], and Dobnikar and Šter [16]. Kolen and Kremer [33] used the concept of recurrent neural networks as advancement over feed forward neural networks. In a closely related description, Hawkins and Boden [25] stated that "as compared to feedforward neural networks recurrent neural networks generally performed better on sequence analysis tasks".…”

Section: Neural Network and Recurrent Neural Networkmentioning

confidence: 99%

“…In principle, recurrent neural networks might be employed by converting the tree into a sequen ce (see, e.g. [25] and [33]). …”

Section: Neural Networkmentioning

confidence: 99%

Parsing with Neural and Finite Automata Networks: A Graph Grammar Approach

Bhargava¹,

Purohit²

2011

IJCA

View full text Add to dashboard Cite

Parsing with finite automata networks implies, in one way, the conversion of a regular expression into a minimal deterministic finite automaton, while parsing with neural networks involves parsing of a natural language sentence. In "Parsing with finite automata networks" finite automata are frequently combined using a set of rules for various operations like union, concatenation, and kleene closure; while in "Parsing with neural networks" an incremental tree is usually obtained, by using a set of rules for connecting a possible parse tree to the previously obtained incremental tree. Apparently, all the above rules that are being applied in parsing whether with finite automata networks or with neural networks belong to some graph transformation rules. These rules depict a new concerned area of grammars known as graph grammar, that is, a grammar that operates on graphs. This research paper presents a twofold investigation on the use of graph grammar as it explores an attempt to use both aspects of graph grammars (to generate a valid language and to parse a language for its validity) for parsing with (i) neural networks and (ii) finite automata networks.

show abstract

“…However the training of complex sequences with long-term dependencies is difficult [3], for example the training of a multiple timescale recurrent neural network (MTRNN) with reasonable network size is computationally intensive, requiring up to one million training epochs for complex sequences [4]. One of the issues for these networks with a large number of parameters is to identify good learning rates for the weight updates during the learning depending on the given problem respective the specific network parameters and the shape of the sequences.…”

Section: Introductionmentioning

confidence: 99%

Adaptive Learning of Linguistic Hierarchy in a Multiple Timescale Recurrent Neural Network

Heinrich

Weber

Wermter

2012

Artificial Neural Networks and Machine Learning – ICANN 2012

View full text Add to dashboard Cite

Abstract. Recent research has revealed that hierarchical linguistic structures can emerge in a recurrent neural network with a sufficient number of delayed context layers. As a representative of this type of network the Multiple Timescale Recurrent Neural Network (MTRNN) has been proposed for recognising and generating known as well as unknown linguistic utterances. However the training of utterances performed in other approaches demands a high training effort. In this paper we propose a robust mechanism for adaptive learning rates and internal states to speed up the training process substantially. In addition we compare the generalisation of the network for the adaptive mechanism as well as the standard fixed learning rates finding at least equal capabilities.

show abstract

A Field Guide to Dynamical Recurrent Networks

Cited by 82 publications

References 0 publications

Action-Driven Perception for a Humanoid

Action-Driven Perception for a Humanoid

Parsing with Neural and Finite Automata Networks: A Graph Grammar Approach

Adaptive Learning of Linguistic Hierarchy in a Multiple Timescale Recurrent Neural Network

Contact Info

Product

Resources

About