Grounding Language Attributes to Objects using Bayesian Eigenobjects

Abstract

We develop a system to disambiguate object instances within the same class based on simple physical descriptions. The system takes as input a natural language phrase and a depth image containing a segmented object and predicts how similar the observed object is to the object described by the phrase. Our system is designed to learn from only a small amount of human-labeled language data and generalize to viewpoints not represented in the language-annotated depth image training set. By decoupling 3D shape representation from language representation, this method is able to ground language to novel objects using a small amount of languageannotated depth-data and a larger corpus of unlabeled 3D object meshes, even when these objects are partially observed from unusual viewpoints. Our system is able to disambiguate between novel objects, observed via depth images, based on natural language descriptions. Our method also enables viewpoint transfer; trained on human-annotated data on a small set of depth images captured from frontal viewpoints, our system successfully predicted object attributes from rear views despite having no such depth images in its training set. Finally, we demonstrate our approach on a Baxter robot, enabling it to pick specific objects based on human-provided natural language descriptions.

Practical Cross-modal Manifold Alignment for Robotic Grounded Language Learning

“…In the robotics domain, Cohen et al [7] combine Bayesian Eigenobjects with a language grounding model that maps natural language phrases and segmented depth images to a shared space. This Bayesian Eigenobjects approach is however evaluated on only three classes of ob-Figure 1: A language + vision manifold alignment approach to language grounding.…”

Talk-to-Resolve: Combining scene understanding and spatial dialogue to resolve granular task ambiguity for a collocated robot

“…Recent works in vision and language navigation [9] and object manipulation [10] can handle complex instructions using multi-modal information, but they still suffer from ambiguity and cascading errors due to misprediction. Although several works have specifically focused on the visual grounding of natural object descriptions [11,12,13], they do not tackle ambiguity and incompleteness using dialogue. Moreover, the predominant approach of end-to-end training for visual grounding is difficult to use in a dialogue system, because the generation of a question pertaining to the instruction, requires a finer-grained understanding of the scene.…”

Language Grounding with 3D Objects

“…Language Grounding in 3D Prior works have associated single-word attributes with 3D object models based on latent representations of 3D meshes [26]. To learn spatial language, vision-andlanguage navigation (VLN) [27] models infer navigation actions from instructions and visual observations in 3D simulated worlds.…”