Vision-based urban navigation procedures for verbally instructed robots

Abstract

The work presented in this thesis is part of a project in instruction based learning (IBL) for mobile robots were a robot is designed that can be instructed by its users through unconstrained natural language. The robot uses vision guidance to follow route instructions in a miniature town model.The aim of the work presented here was to detenn.ine the functional vocabulary of the robot in d1e form of "primitive procedures". In contrast to previous work in the field of instructable robots this was done following a "user-centred" approach were the main concern was to create primitive procedures that can be directly associated with natural language instructions. To achieve this, a corpus of human-to-human natural language instructions was collected and analysed. A set of primitive actions was found with which the collected corpus could be represented. These primitive actions were then implemented as robot-executable procedures.Natural language instructions are under-specified when destined to be executed by a robot. This is because instructors omit information tl1at they consider as "commonsense" and rely on the listener's sensory-motor capabilities to determine the details of the task execution. In this thesis the underspecification problem is solved by determining the missing information, either during the learning of new routes or during their execution by the robot. During learning, the missing information is determined by imitating the commonsense approach human listeners take to achieve the same purpose. During execution, missing information, such as the location of road layout features mentioned in route instructions, is determined from the robot's view by using image template matching. The original contribution of this thesis, in both these methods, lies in the fact that they are driven by the natural language examples found in the corpus collected for the IDL project. 3During the testing phase a high success rate of primitive calls, when these were considered individually, showed that the under-specification problem has overall been solved. A novel method for testing the primitive procedures, as part of complete route descriptions, is also proposed in this thesis. This was done by comparing the performance of human subjects when driving the robot, following route descriptions, with the performance of the robot when executing the same route descriptions. The results obtained from this comparison clearly indicated where errors occur from the time when a human speaker gives a route description to the time when the task is executed by a human listener or by the robot.Finally, a software speed controller is proposed in this thesis in order to control the wheel speeds of the robot used in this project. The controller employs PI (Proportional and Integral) and PID (Proportional, Integral and Differential) control and provides a good alternative to expensive hardware.

Spoken Language and Vision for Adaptive Human-Robot Cooperation

“…Based on an analysis of the resulting speech corpora, they identified a set of verbal action chunks that could map onto robot control primitives. More recently, they demonstrated the effectiveness of navigation instructions translated into these primitive procedures for actual robot navigation (Kyriacou et al 2005). This research indicates the importance of implementing the mapping between language and behavioural primitives for high-level natural language instruction or programming.…”

“…In this context, two domains of interaction that humans exploit with great fidelity are spoken language, and the visual ability to observe and understand intentional action. A good deal of research effort has been dedicated to the specification and implementation of spoken language systems for human-robot interaction (Crangle & Suppes 1994, Lauria et al 2002, Severinson-Eklund 2003, Kyriacou et al 2005, Mavrides & Roy 2006. The research described in the current chapter extends these approaches with a Spoken Language Programming system that allows a more detailed specification of conditional execution, and by using language as a compliment to vision-based action perception as a mechanism for indicating how things are to be done, in the context of cooperative, turn-taking behavior.…”

Multimodal cognitive interface for robot navigation

“…Many research efforts focus on using spatial language to control the robot's position and behavior, or to enable it to answer questions about what it senses. In Lauria et al (2001), Kyriacou et al (2002), Bugmann et al (2004), InstructionBased Learning (IBL) is built to train mobile robots using natural language instruction to describe a navigate task. In this project, a robot is instructed on how to travel from one place to another in a miniature town.…”

Robot Navigation based on the Mapping of Coarse Qualitative Route Descriptions to Route Graphs

“…Kyriacou et.al. describe in [6] a verbally instructed robot that executes a given route description in an artificial miniature town. Encouraged by corpora analysis they transfer a given route description into a list of action chunks accompanied by spatial relations and landmarks.…”