A natural language planner interface for mobile manipulators

Abstract

Natural language interfaces for robot control aspire to find the best sequence of actions that reflect the behavior intended by the instruction. This is difficult because of the diversity of language, variety of environments, and heterogeneity of tasks. Previous work has demonstrated that probabilistic graphical models constructed from the parse structure of natural language can be used to identify motions that most closely resemble verb phrases. Such approaches however quickly succumb to computational bottlenecks imposed by construction and search the space of possible actions. Planning constraints, which define goal regions and separate the admissible and inadmissible states in an environment model, provide an interesting alternative to represent the meaning of verb phrases. In this paper we present a new model called the Distributed Correspondence Graph (DCG) to infer the most likely set of planning constraints from natural language instructions. A trajectory planner then uses these planning constraints to find a sequence of actions that resemble the instruction. Separating the problem of identifying the action encoded by the language into individual steps of planning constraint inference and motion planning enables us to avoid computational costs associated with generation and evaluation of many trajectories. We present experimental results from comparative experiments that demonstrate improvements in efficiency in natural language understanding without loss of accuracy.

Proceedings of the First Workshop on Language Grounding for Robotics

“…For example, a number of research-ers have built systems to satisfy the need for grounded language in a robot, including Steels and Hild (2012), Tellex et al (2014), and Eppe et al (2016). However, none of those systems conforms to the three aspects of a cognitive modeling approach outlined above.…”

“…Previous work has approached the grounding problem using a variety of resources and approaches, for instance, either using annotated visual datasets (Silberer and Lapata, 2014;Socher et al, 2014;Naim et al, 2015;Al-Omari et al, 2016;Tellex et al, 2014;Matuszek et al, 2012Matuszek et al, , 2014, or through interactions with other agents or real humans (Kollar et al, 2013;Tellex et al, 2013;Thomason et al, 2015Thomason et al, , 2016Skocaj et al, 2016;Yu et al, 2016c), where feedback from other agents is used to learn new concepts.…”

From Treebank Parses to Episodic Logic and Commonsense Inference

“…Impressive thematic scope is achieved in (Berant et al 2013, Kwiatkowski et al 2013), but the target semantic language (for Freebase access) is still restricted to database operations such as join, intersection, and set cardinality. Another popular domain is command execution by robots (e.g., Tellex 2011, Howard et al 2013.…”

Proceedings of the ACL 2014 Workshop on Semantic Parsing

“…Impressive thematic scope is achieved in ), but the target semantic language (for Freebase access) is still restricted to database operations such as join, intersection, and set cardinality. Another popular domain is command execution by robots (e.g., Tellex 2011, Howard et al 2013.…”

“…Recently we have seen an explosion of work on learning semantic parsers (e.g., Matuszek, et al, 2012;Tellex et al, 2013;Branavan et al, 2010, Chen et al, 2011). While such work shows promise, the results are highly domain dependent and useful only for that domain.…”