Interactive natural language acquisition in a multi-modal recurrent neural architecture

Abstract

For the complex human brain that enables us to communicate in natural language, we gathered good understandings of principles underlying language acquisition and processing, knowledge about sociocultural conditions, and insights into activity patterns in the brain. However, we were not yet able to understand the behavioural and mechanistic characteristics for natural language and how mechanisms in the brain allow to acquire and process language. In bridging the insights from behavioural psychology and neuroscience, the goal of this paper is to contribute a computational understanding of appropriate characteristics that favour language acquisition. Accordingly, we provide concepts and refinements in cognitive modelling regarding principles and mechanisms in the brain and propose a neurocognitively plausible model for embodied language acquisition from real-world interaction of a humanoid robot with its environment. In particular, the architecture consists of a continuous time recurrent neural network, where parts have different leakage characteristics and thus operate on multiple timescales for every modality and the association of the higher level nodes of all modalities into cell assemblies. The model is capable of learning language production grounded in both, temporal dynamic somatosensation and vision, and features hierarchical concept abstraction, concept decomposition, multi-modal integration, and self-organisation of latent representations. ARTICLE HISTORY

Building brains that communicate like machines

“…We argue that these conditions inherently enable the development of distributed representations of knowledge. For example, in our research, we found that architectural mechanisms, like different timings in the information processing in the cortex, foster compositionality that in turn enables both the development of more complex body actions and the development of language competence from primitives (Heinrich 2016). These kinds of distributed representations are coherent with the cognitive science on embodied cognition.…”

Sensorimotor input as a language generalisation tool: a neurorobotics model for generation and generalisation of noun-verb combinations with sensorimotor inputs

“…For instance, in Sugita and Tani (2005), such association learning occurring on the PB level binds the semantic and the behaviour representations. Similar association learning also can be found in Heinrich and Wermter (2018). On the other hand, the single RNN we use, although with more complexity in training, allows a higher generalisation abilities because all the modalities are learnt in a single dynamical system.…”

Symbol Emergence in Cognitive Developmental Systems: A Survey

“…The central idea is that continuous neural computation and predictive coding of sensorimotor information is crucial for internal representation formation. Recently, we can find many studies that apply deep learning scheme to multimodal human-robot interaction including linguistic modality [124]- [128]. Recent studies on artificial general intelligence based on DRL follow a similar idea.…”