Chained learning architectures in a simple closed-loop behavioural context

Kulvičius, Tomas; Porr, Bernd; Wörgötter, Florentin

doi:10.1007/s00422-007-0176-y

Cited by 13 publications

(13 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Learning, storing, inferring and executing sequences is a key topic in experimental [71]–[79], and theoretical neurosciences [80]–[82]; and robotics [83]–[86]. An early approach to modelling sequence processing focussed on feed-forward architectures.…”

Section: Discussionmentioning

confidence: 99%

“…This requires the ‘dynamic fusion’ of bottom-up sensory input and top-down predictions, Several authors e.g., [83], [89]–[92] use recurrent networks to implement this fusion. Exact Bayesian schemes based on discrete hierarchical hidden Markov models, specified as a temporal hierarchy, have been used to implement memory and recognition [93].…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Recognizing Sequences of Sequences

et al. 2009

View full text Add to dashboard Cite

The brain's decoding of fast sensory streams is currently impossible to emulate, even approximately, with artificial agents. For example, robust speech recognition is relatively easy for humans but exceptionally difficult for artificial speech recognition systems. In this paper, we propose that recognition can be simplified with an internal model of how sensory input is generated, when formulated in a Bayesian framework. We show that a plausible candidate for an internal or generative model is a hierarchy of ‘stable heteroclinic channels’. This model describes continuous dynamics in the environment as a hierarchy of sequences, where slower sequences cause faster sequences. Under this model, online recognition corresponds to the dynamic decoding of causal sequences, giving a representation of the environment with predictive power on several timescales. We illustrate the ensuing decoding or recognition scheme using synthetic sequences of syllables, where syllables are sequences of phonemes and phonemes are sequences of sound-wave modulations. By presenting anomalous stimuli, we find that the resulting recognition dynamics disclose inference at multiple time scales and are reminiscent of neuronal dynamics seen in the real brain.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Recognizing Sequences of Sequences

et al. 2009

View full text Add to dashboard Cite

show abstract

“…Both rules are stable and stability for x 0 = 0 can be mathematically proved for both rules even when using filter banks ( Porr and Wörgötter 2006 , 2007 ). These rules have now been successfully tested in a variety of different applications ( Porr and Wörgötter 2006 ; Kolodziejski et al 2006 , 2007 ; Manoonpong et al 2007 ) and even chains of learning neurons can be constructed in a convergent way ( Kulvicius et al 2007 ).…”

Section: Results When Using a Filter Bankmentioning

confidence: 99%

Mathematical properties of neuronal TD-rules and differential Hebbian learning: a comparison

2008

Self Cite

View full text Add to dashboard Cite

A confusingly wide variety of temporally asymmetric learning rules exists related to reinforcement learning and/or to spike-timing dependent plasticity, many of which look exceedingly similar, while displaying strongly different behavior. These rules often find their use in control tasks, for example in robotics and for this rigorous convergence and numerical stability is required. The goal of this article is to review these rules and compare them to provide a better overview over their different properties. Two main classes will be discussed: temporal difference (TD) rules and correlation based (differential hebbian) rules and some transition cases. In general we will focus on neuronal implementations with changeable synaptic weights and a time-continuous representation of activity. In a machine learning (non-neuronal) context, for TD-learning a solid mathematical theory has existed since several years. This can partly be transfered to a neuronal framework, too. On the other hand, only now a more complete theory has also emerged for differential Hebb rules. In general rules differ by their convergence conditions and their numerical stability, which can lead to very undesirable behavior, when wanting to apply them. For TD, convergence can be enforced with a certain output condition assuring that the δ-error drops on average to zero (output control). Correlation based rules, on the other hand, converge when one input drops to zero (input control). Temporally asymmetric learning rules treat situations where incoming stimuli follow each other in time. Thus, it is necessary to remember the first stimulus to be able to relate it to the later occurring second one. To this end different types of so-called eligibility traces are being used by these two different types of rules. This aspect leads again to different properties of TD and differential Hebbian learning as discussed here. Thus, this paper, while also presenting several novel mathematical results, is mainly meant to provide a road map through the different neuronally emulated temporal asymmetrical learning rules and their behavior to provide some guidance for possible applications.

show abstract

“…This has the advantage that the error e j can directly be identified with the reflex action (see Figure 1), so that v k by design then executes both the reflex and the learned/adaptive actions. However, because the error signal e j and learned signals v j are mixed together at v k , information is lost which cannot be recovered for deeper layers, thereby restricting learning to shallow network architectures (Kulvicius, Porr, & Wo¨rgo¨tter, 2007). This also means that both error and learned signals need to have a direct behavioural meaning because both will eventually cause a behavioural output.…”

Section: Discussionmentioning

confidence: 99%

Forward propagation closed loop learning

Porr¹,

Miller²

2019

Adaptive Behavior

Self Cite

View full text Add to dashboard Cite

For an autonomous agent, the inputs are the sensory data that inform the agent of the state of the world, and the outputs are their actions, which act on the world and consequently produce new sensory inputs. The agent only knows of its own actions via their effect on future inputs; therefore desired states, and error signals, are most naturally defined in terms of the inputs. Most machine learning algorithms, however, operate in terms of desired outputs. For example, backpropagation takes target output values and propagates the corresponding error backwards through the network in order to change the weights. In closed loop settings, it is far more obvious how to define desired sensory inputs than desired actions, however. To train a deep network using errors defined in the input space would call for an algorithm that can propagate those errors forwards through the network, from input layer to output layer, in much the same way that activations are propagated. In this article, we present a novel learning algorithm which performs such ‘forward-propagation’ of errors. We demonstrate its performance, first in a simple line follower and then in a 1st person shooter game.

show abstract

Chained learning architectures in a simple closed-loop behavioural context

Cited by 13 publications

References 41 publications

Recognizing Sequences of Sequences

Recognizing Sequences of Sequences

Mathematical properties of neuronal TD-rules and differential Hebbian learning: a comparison

Forward propagation closed loop learning

Contact Info

Product

Resources

About