Multi-modal human-machine communication for instructing robot grasping tasks

Abstract

A major challenge for the realization of intelligent robots is to supply them with cognitive abilities in order to allow ordinary users to program them easily and intuitively. One way of such programming is teaching work tasks by interactive demonstration. To make this effective and convenient for the user, the machine must be capable to establish a common focus of attention and be able to use and integrate spoken instructions, visual perceptions, and non-verbal clues like gestural commands. We report progress in building a hybrid architecture that combines statistical methods, neural networks, and finite state machines into an integrated system for instructing grasping tasks by man-machine interaction. The system combines the GRAVIS-robot for visual attention and gestural instruction with an intelligent interface for speech recognition and linguistic interpretation, and an modality fusion module to allow multi-modal task-oriented man-machine communication with respect to dextrous robot manipulation of objects.

Platform portable anthropomorphic grasping with the bielefeld 20-DOF shadow and 9-DOF TUM hand

“…The GRAVIS robot system [20] combines visual attention and gestural instruction with an intelligent interface for speech recognition and linguistic interpretation to allow multi-modal task-oriented instructions. For manipulation tasks this setup employs a standard 6-DOF PUMA manipulator operated with the real-time RCCL-command library together with a 9-DOF dextrous robot hand developed at the Technical University of Munich (TUM).…”

“…The implementations of step 1 of our grasp strategy differ between the two robot hand setups in use. The TUM Hand setup allows the human instructor to identify an object to be grasped by speech, pointing gestures, or a combination thereof [20]. The 3D position of the referred object is resolved by a stereo vision system to an accuracy of about 3 cm.…”

“…Consequently, we have previously proposed to enable imitation grasping [18] in the context of a long-term research project [19] aiming at the realization of a robot system that is instructable by speech and gesture, has visual capabilities, attentive behavior, and can execute grasping actions [20], [21] (see Fig. 1).…”

A Dual Interaction Perspective for Robot Cognition: Grasping as a “Rosetta Stone”

“…Starting from a previous robotics setup developed in the course of the special research unit "SFB 360" and providing a large number of specialised processing modules [20], we have implemented a robot system ( fig. 8) whose grasping abilities connect sensori-motor control with vision and language understanding.…”

Towards Grasp-Oriented Visual Perception for Humanoid Robots

“…Within this domain, McGuire et al 3 for example have developed a system for attending to pointing hands in reference to objects of interest. First, the robot's vision system focuses on human hand gestures (i.e.…”