Language-Guided Adaptive Perception for Efficient Grounded Communication with Robotic Manipulators in Cluttered Environments

Abstract

The utility of collaborative manipulators for shared tasks is highly dependent on the speed and accuracy of communication between the human and the robot. The run-time of recently developed probabilistic inference models for situated symbol grounding of natural language instructions depends on the complexity of the representation of the environment in which they reason. As we move towards more complex bi-directional interactions, tasks, and environments, we need intelligent perception models that can selectively infer precise pose, semantics, and affordances of the objects when inferring exhaustively detailed world models is inefficient and prohibits real-time interaction with these robots. In this paper we propose a model of language and perception for the problem of adapting the configuration of the robot perception pipeline for tasks where constructing exhaustively detailed models of the environment is inefficient and inconsequential for symbol grounding. We present experimental results from a synthetic corpus of natural language instructions for robot manipulation in example environments. The results demonstrate that by adapting perception we get significant gains in terms of run-time for perception and situated symbol grounding of the language instructions without a loss in the accuracy of the latter.

Inferring Compact Representations for Efficient Natural Language Understanding of Robot Instructions

“…Table II shows the impact of adaptive perception on the compactness of the approximated world representations. Consistent with previous evaluations [20], reducing the cardinality of the world model improves the runtime of language grounding. Figure 3 demonstrates the impact of adaptive perception for the example instruction "drive to the nearest cup in the kitchen."…”

“…In previous experiments [20], we observed that language grounding was faster in environments inferred by adaptive perception than non-adaptive perception. Also the adaptive perception was found to be faster than its counterpart.…”

“…compact representations can reduce the runtime requirements of robot perception. As we have seen [20], the reduction in runtime is proportional to the sparsity of classifiers necessary to extract a sufficient detailed environment model that is suitable for the grounding of specific instructions.…”

“…This model extracts characteristics of the representation from free-form utterances to "lazily" reason over the small subset of available knowledge pertinent to the task. Specifically, we build upon recent work on adapting perception pipelines from natural language instructions [20] to infer subsets of observations that we use to construct instruction-specific representations of the environment. These induced representations are more efficient to search, yet still express the correct hierarchies and affordances necessary to perform the task.…”

Language Understanding for Field and Service Robots in a Priori Unknown Environments

“…The approach to natural language understanding of robot instructions in this paper relies on variations of the Distributed Correspondence Graph (DCG) [3,4,10,15,17,42,45,75,79,80]. The DCG and the Hierarchical Distributed Correspondence Graph (HDCG) [22], Adaptive Distributed Correspondence Graph (ADCG) [82], and Hierarchical Adaptive Distributed Correspondence Graph (HADCG) [83] variations of the model formulate the problem of natural language understanding as probabilistic inference in a factor graph using models to approximate the conditional probabilities of a correspondence variable φ in the context of language Λ, symbols Γ, and the environment model S from corpora of annotated examples.…”