Building a multimodal human-robot interface

Abstract

However, the situation becomes a bit more complex when we begin to build and interact with machines or robots that either look like humans or have human functionalities and capabilities. Then, people well might interact with their humanlike machines in ways that mimic humanhuman communication.For example, if a robot has a face, a human might interact with it similarly to how humans interact with other creatures with faces. Specifically, a human might talk to it, gesture to it, smile at it, and so on. If a human interacts with a computer or a machine that understands spoken commands, the human might converse with the machine, expecting it to have competence in spoken language.In our research on a multimodal interface to mobile robots, we have assumed a model of communication and interaction that, in a sense, mimics how people communicate. Our interface therefore incorporates both natural language understanding and gesture recognition as communication modes. We limited the interface to these two modes to simplify integrating them in the interface and to make our research more tractable.We believe that with an integrated system, the user is less concerned with how to communicate (which interactive mode to employ for a task) and is therefore free to concentrate on the tasks and goals at hand. Because we integrate all our system's components, users can choose any combination of our interface's modalities. The onus is on our interface to integrate the input, process it, and produce the desired results. RequirementsAs developers of speech recognition and naturallanguage-understanding systems no doubt know, humans expect a fairly sophisticated level of recognition, understanding, and interaction. Speech systems that limit the human to simple utterances, prescribed formulaic utter-

Assisted Teleoperation Strategies for Aggressively Controlling a Robot Arm with 2D Input

“…Nontraditional input devices such as voice [12] and gesture [19] are of interest for novice robot control as well as in assistive robotics for those who have lost function or fine motor skills. Another technology under consideration is braincomputer interfaces (BCIs).…”

Designing Laser Gesture Interface for Robot Control

“…Perzanowski et al provided multimodal interface in which the user can specify the target location by tapping a map on a PDA screen [4]. Lundberg et al also proposed a PDA interface for a robot, where the user can specify the area for the robot to explore [5].…”

Multimodal Command Language to Direct Home-Use Robots

“…Thus, a well-defined multimodal command set combining verbal and nonverbal messages would help users of home-use robots. Perzanowski et al developed a multimodal human-robot interface that enables users to give commands combining spoken commands and pointing gestures (Perzanowski et al, 2001). In the system, spoken commands are analysed using a speech-to-text system and a natural language understanding system that parses text strings.…”