Analysing the Multiple Timescale Recurrent Neural Network for Embodied Language Understanding

Abstract

How the human brain understands natural language and how we can exploit this understanding for building intelligent grounded language systems is open research. Recently, researchers claimed that language is embodied in most -if not all -sensory and sensorimotor modalities and that the brain's architecture favours the emergence of language. In this chapter we investigate the characteristics of such an architecture and propose a model based on the Multiple Timescale Recurrent Neural Network, extended by embodied visual perception, and tested in a real world scenario. We show that such an architecture can learn the meaning of utterances with respect to visual perception and that it can produce verbal utterances that correctly describe previously unknown scenes. In addition we rigorously study the timescale mechanism (also known as hysteresis) and explore the impact of the architectural connectivity in the language acquisition task.

Interactive natural language acquisition in a multi-modal recurrent neural architecture

“…Although the model was reproducing learned symbolic sentences quite well in their experiments, generalisation was not possible to test, because the generation of sentences was initiated by the internal state of the Csc units, which had to be trained individually for every sentence in the model. Heinrich, Magg, and Wermter (2015) extended this model to process the language embodied in a way that visual input will trigger the model to produce a meaningful verbal utterance that appropriately represents the input. The architecture, called EMBMTRNN model, consists of similar MTRNN layers for the language network, where a verbal utterance is processed as a sequence on phoneme level based on initial activity on an overall concept level.…”

Sensorimotor input as a language generalisation tool: a neurorobotics model for generation and generalisation of noun-verb combinations with sensorimotor inputs

“…The neural dynamics in our MTRNN exhibited a dynamics which are different from those reported in Hinoshita et al (2009) and Heinrich et al (2015). Whereas the noun (or object perceptual inputs) play a significant factor in the dynamics of context layers in these two examples, our network has minimised the effects of nouns or the object perception.…”

“…They also have similar information bi-directional flows which allow the networks to recognise and to generate the time sequences. Despite their similarities, compared with LSTM, the MTRNN have other distinct features: First, from the above experiments and from other MTRNN experiments (Heinrich et al 2015;Hinoshita et al 2009), it has been shown that the fast context layers and slow context layers exhibit various dynamics to explicitly represent the relationship between the verbs and nouns. The deep LSTM, on the contrary, has not been reported to have similar dynamics.…”

“…Although similar RNNPB (Sugita and Tani 2005) or MTRNN (Heinrich et al 2015) networks have been used to learn verbs and nouns features with motor actions and visual features, the model we will use is a single MTRNN model to learn both the sensory and motor information in a single set of sequences, because we regard the perception and action having inseparable links (e.g. Wolpert et al 1995;Noë 2001) and should be encoded solely as similar data structures.…”

“…To summarise, compared with the connectionist models on semantic compositionality (Sugita and Tani 2005;Heinrich et al 2015), the novelties of our model and experiments are: -Instead of using the neural binding methods on multiple RNNs, the hierarchical MTRNN provides another perspective to model the emergence of semantic compositionality over multi-modal data, which may be more parallel to the perception-action coupling of different levels of the nervous system (Sperry 1952): perception and action processes are functionally intertwined, which we represent in the recurrent connections from the low to the top layer in our hierarchical network. -Technically, in our model, the multi-modal data (language, visual and proprioceptive) was implemented into a single hierarchical network.…”