The Coordinating Role of Language in Real-Time Multimodal Learning of Cooperative Tasks

Abstract

One of the defining characteristics of human cognition is our outstanding capacity to cooperate. A central requirement for cooperation is the ability to establish a "shared plan" -which defines the interlaced actions of the two cooperating agents -in real time, and even to negotiate this shared plan during its execution.In the current research we identify the requirements for cooperation, extending our earlier work in this area. These requirements include the ability to negotiate a shared plan using spoken language, to learn new component actions within that plan, based on visual observation and kinesthetic demonstration, and finally to coordinate all of these functions in real time. We present a cognitive system that implements these requirements, and demonstrate the system's ability to allow a Nao humanoid robot to learn a non-trivial cooperative task in real-time. We further provide a concrete demonstration of how the real-time learning capability can be easily deployed on different platform, in this case the iCub humanoid. The results are considered in the context of how the development of language in the human infant provides a powerful lever in the development of cooperative plans from lower-level sensorimotor capabilities.Index Terms-cooperation, humanoid robot, spoken language interaction, shared plans, situated and social learning.

Coordination mechanism for integrated design of Human-Robot Interaction scenarios

“…Reinforcement learning has also been used to coordinate execution of lower-level behaviors [27] using a topological map as a task-relevant state space instead of using the sensory space of the robot. DAC-h3 [28], based on the Distributed Adaptive Control (DAC) architecture [29,30], proposes a principled organization of various functional modules into a biologically grounded, multi-layered cognitive architecture, with behaviors which take care of motor control and regulate the internal drives or needs of the robot. HAMMER [31] is a computational architecture that provides top-down control of visual attention for computing resource allocation in the specific application for observing action demonstrations while learning skills.…”

An integrated probabilistic framework for robot perception, learning and memory

“…These cameras are mounted in the head of the robot, in positions similar to where eyes are located on a human face. This configuration, together with appropriate computer vision algorithms, enables the recognition and tracking of regions of interests from the scene [14]. Physical and safe interaction of the robot with its environment is possible with the use of its tactile sensory system that, built with a capacitive technology, provides pressure measurements in [0 255] sampled at 50 Hz [15], [16].…”

Machines Learning - Towards a New Synthetic Autobiographical Memory

“…for maximum likelihood estimation [24]), or (ii) annotated high-level sequences (e.g. of social interactions [35]). The initial problem with approach (i) -storage capacity -is being alleviated to a degree by improvements in hard-drive capacity.…”

“…Recent success has been had in demonstrating the effectiveness of approach (ii) such as in cooperative tasks [35], or learning through social interaction [36]. These models are largely symbolic, similar in spirit to classic models of cognition like ACT-R [1] and others (reviewed in [52]), with hand-set higher level representations and action scripts coordinated into event memories [37].…”