An Experiment on Behavior Generalization and the Emergence of Linguistic Compositionality in Evolving Robots

Abstract

Populations of simulated agents controlled by dynamical neural networks are trained by artificial evolution to access linguistic instructions and to execute them by indicating, touching, or moving specific target objects. During training the agent experiences only a subset of all object/action pairs. During postevaluation, some of the successful agents proved to be able to access and execute also linguistic instructions not experienced during training. This owes to the development of a semantic space, grounded in the sensory motor capability of the agent and organized in a systematized way in order to facilitate linguistic compositionality and behavioral generalization. Compositionality seems to be underpinned by a capability of the agents to access and execute the instructions by temporally decomposing their linguistic and behavioral aspects into their constituent parts (i.e., finding the target object and executing the required action). The comparison between two experimental conditions, in one of which the agents are required to ignore rather than to indicate objects, shows that the composition of the behavioral set significantly influences the development of compositional semantic structures.Non peer reviewe

Imitating operations on internal cognitive structures for language aquisition

“…The task solved in the presented experiment is close to the task solved in [10]. There are also structural similarities in how the task is solved since the architecture presented in [10] is not based on separate systems for language and action.…”

“…Furthermore the proposed architecture does not use neural networks. The generalization ability exhibited by the system in [10] is also exhibited by the proposed algorithm which is able to respond properly to novel combinations of linguistic commands. See [11] for another artificial neural network based approach to this type of task.…”

“…There are also structural similarities in how the task is solved since the architecture presented in [10] is not based on separate systems for language and action. The difference between the proposed architecture and [10] is the use of imitation learning methods instead of rewards and the fact that the proposed architecture does not use a symbolic representation of the communicative acts. Furthermore the proposed architecture does not use neural networks.…”

“…See [11] for another artificial neural network based approach to this type of task. However, in [11] the action and linguistic units are separate units and, like in [10], the linguistic inputs are symbolic.…”

Learning semantic components from subsymbolic multimodal perception

“…For example, Sugita and Tani [6] have built a recurrent neural networks which is capable of discovering a joint grammar based on simple object-action pairs. Similarly, Tuci et al [7] trained artificial agents on executing actions corresponding to linguistic instructions that are object-action pairs. They show how the joint languageaction representation allows the agents to generalize action knowledge to unknown object-action pairs.…”