Toward understanding natural language directions

Abstract

Abstract-Speaking using unconstrained natural language is an intuitive and flexible way for humans to interact with robots. Understanding this kind of linguistic input is challenging because diverse words and phrases must be mapped into structures that the robot can understand, and elements in those structures must be grounded in an uncertain environment. We present a system that follows natural language directions by extracting a sequence of spatial description clauses from the linguistic input and then infers the most probable path through the environment given only information about the environmental geometry and detected visible objects. We use a probabilistic graphical model that factors into three key components. The first component grounds landmark phrases such as "the computers" in the perceptual frame of the robot by exploiting co-occurrence statistics from a database of tagged images such as Flickr. Second, a spatial reasoning component judges how well spatial relations such as "past the computers" describe a path. Finally, verb phrases such as "turn right" are modeled according to the amount of change in orientation in the path. Our system follows 60% of the directions in our corpus to within 15 meters of the true destination, significantly outperforming other approaches.

Learning to interpret spatial natural language in terms of qualitative spatial relations*

“…The kind of spatial language interpretation should be particularly applicable in several domains, whenever a mapping from language to abstract spatial representations are required. Examples are dialogue systems (Kruijff et al, 2007), spatial assistance system for design or media production (Bhatt et al, this volume), text-to-scene conversion systems (Rouhizadeh et al, 2011), virtual human cooperations (Nguyen and Wachsmuth, this volume), and robotic navigation instructions (Kollar et al, 2010).…”

“…This allows us to formally structure and present spatial information based on available linguistic characteristics and to deal with aspects of ambiguity and uncertainty caused by linguistic underspecification, granularity differences, context dependencies, metaphors, etc. This is particularly important as we 6 apply our approach to arbitrary linguistic sources, which is in contrast to using only a limited range of spatial language or phrases of which it is known that they contain spatial information (Kollar et al, 2010;Kelleher and Costello, 2009;Li et al, 2007).…”

“…It focuses on fable stories, however, it only supports limited automatic recognition of spatial elements. The system presented in (Kollar et al, 2010) operates in a human-robot interaction context and follows natural language route directions. It is more flexible in extracting spatial information, though it expects only spatial language as input.…”

Grounding spatial relations in natural language by fuzzy representation for human-robot interaction

“…A command in natural language can be decomposed into a hierarchy of spatial description clauses. Each of the clauses consists of a subject, a verb for the action to take, a spatial relation that describes the relation between the subject and the reference object, and a landmark that is referenced by the relation [2]. The groundings of the linguistic constituents can be inferred from a grounding graph -a probabilistic graph that was trained on a corpus of natural language commands paired with groundings for each part of the command [3].…”

“…Within the context of robot control and human-robot interaction, the aim of natural language parsing has been to derive a formal representation from natural language utterances or commands. The representation could be obtained through observation [1], by parsing the sentences into spatial description clauses (SDCs) [2,3], or through learning [4]. In comparison with the natural language parsing problem, the options for the solution of the natural language grounding problem are rather limited.…”