Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction

Misra, Dipendra; Bennett, Andrew F.; Blukis, Valts; Niklasson, Eyvind; Shatkhin, Max; Artzi, Yoav

doi:10.48550/arxiv.1809.00786

Cited by 13 publications

(21 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[52] tackle 'interactive navigation', where the robot can bump into and push objects during navigation, but does not have an arm. Some works [56][57][58] abstract away gross motor control entirely by using symbolic interaction capabilities (e.g. a 'pick up X' action) or a 'magic pointer' [9].…”

Section: Related Workmentioning

confidence: 99%

Habitat 2.0: Training Home Assistants to Rearrange their Habitat

Szot¹,

Clegg²,

Undersander³

et al. 2021

Preprint

View full text Add to dashboard Cite

We introduce Habitat 2.0 (H2.0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios. We make comprehensive contributions to all levels of the embodied AI stack -data, simulation, and benchmark tasks. Specifically, we present: (i) ReplicaCAD: an artist-authored, annotated, reconfigurable 3D dataset of apartments (matching real spaces) with articulated objects (e.g. cabinets and drawers that can open/close); (ii) H2.0: a high-performance physics-enabled 3D simulator with speeds exceeding 25,000 simulation steps per second (850× real-time) on an 8-GPU node, representing 100× speed-ups over prior work; and, (iii) Home Assistant Benchmark (HAB): a suite of common tasks for assistive robots (tidy the house, prepare groceries, set the table) that test a range of mobile manipulation capabilities. These large-scale engineering contributions allow us to systematically compare deep reinforcement learning (RL) at scale and classical sense-plan-act (SPA) pipelines in long-horizon structured tasks, with an emphasis on generalization to new objects, receptacles, and layouts. We find that (1) flat RL policies struggle on HAB compared to hierarchical ones; (2) a hierarchy with independent skills suffers from 'hand-off problems', and (3) SPA pipelines are more brittle than RL policies. Figure 1: A mobile manipulator (Fetch robot) simulated in Habitat 2.0 performing rearrangement tasks in a ReplicaCAD apartment -(left) opening a drawer before picking up an item from it, and (right) placing an object into the bowl after navigating to the table. Best viewed in motion at https://sites.google.com/view/habitat2. Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Habitat 2.0: Training Home Assistants to Rearrange their Habitat

Szot¹,

Clegg²,

Undersander³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Natural language, the most common modality for human-human and human-robot communication, can realize the grounding in various ways. For communication with robots, language can be interpreted from instructional commands to actions [29,30]. For static images or texts, it can be either visually grounded [31,32] or text-based [33] Q&A.…”

Section: Language Groundingmentioning

confidence: 99%

Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scene

Wu¹,

Wu²,

Zhu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Human-robot collaboration is an essential research topic in artificial intelligence (AI), enabling researchers to devise cognitive AI systems and affords an intuitive means for users to interact with the robot. Of note, communication plays a central role. To date, prior studies in embodied agent navigation have only demonstrated that human languages facilitate communication by instructions in natural languages. Nevertheless, a plethora of other forms of communication is left unexplored. In fact, human communication originated in gestures and oftentimes is delivered through multimodal cues, e.g., "go there" with a pointing gesture. To bridge the gap and fill in the missing dimension of communication in embodied agent navigation, we propose investigating the effects of using gestures as the communicative interface instead of verbal cues. Specifically, we develop a VR-based 3D simulation environment, named Gesture-based THOR (Ges-THOR), based on AI2-THOR platform. In this virtual environment, a human player is placed in the same virtual scene and shepherds the artificial agent using only gestures. The agent is tasked to solve the navigation problem guided by natural gestures with unknown semantics; we do not use any predefined gestures due to the diversity and versatile nature of human gestures. We argue that learning the semantics of natural gestures is mutually beneficial to learning the navigation task-learn to communicate and communicate to learn. In a series of experiments, we demonstrate that human gesture cues, even without predefined semantics, improve the object-goal navigation for an embodied agent, outperforming various state-of-the-art methods.

show abstract

“…End to end learning approaches Several recent deep learning approaches propose to learn a mapping directly from inputs to actions, whether structured observations are provided [22,33] or the agent deals with raw visual observations [25,43]. Cross modal grounding of language instructions to visual observations is often used in several works, via e.g., reinforcement learning [38,37], autoencoder architectures that impose a language instructions-based heat map on the visual observations (using U-net architectures [24], attention mechanisms [49], or implementation of non linear differentiable filters [2]). However, as we show later in results, going end to end may not be best for generalizing to perform compositional tasks in unseen environments.…”

Section: Related Workmentioning

confidence: 99%

“…Using such keywords from language to identify regions in image is known as referring expression image segmentation (e.g.- [44]). We propose a simple restructuring of language inputs to LingUNet [24] and retrain it conditioned on semantic word labels provided by the language understanding module.…”

Section: Overview Of Movilan Frameworkmentioning

confidence: 99%

“…A technique that segments images based on key words in a natural language expression is known as referring expression image segmentation. We adapt the LingUNet learning framework [24] for predicting a probability distribution over input pixels of RGB image. We take a slight departure from the original formulation for presenting instruction encodings to a UNet architecture [29].…”

Section: Vision Language Groundingmentioning

confidence: 99%

See 1 more Smart Citation

A modular vision language navigation and manipulation framework for long horizon compositional tasks in indoor environment

Saha¹,

Fotouhif²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper we propose a new framework -MoVi-Lan (Modular Vision and Language) for execution of visually grounded natural language instructions for day to day indoor household tasks. While several data-driven, endto-end learning frameworks have been proposed for targeted navigation tasks based on the vision and language modalities, performance on recent benchmark data sets revealed the gap in developing comprehensive techniques for long horizon, compositional tasks (involving manipulation and navigation) with diverse object categories, realistic instructions and visual scenarios with non-reversible state changes. We propose a modular approach to deal with the combined navigation and object interaction problem without the need for strictly aligned vision and language training data (e.g., in the form of expert demonstrated trajectories). Such an approach is a significant departure from the traditional end-to-end techniques in this space and allows for a more tractable training process with separate vision and language data sets. Specifically, we propose a novel geometry-aware mapping technique for cluttered indoor environments, and a language understanding model generalized for household instruction following. We demonstrate a significant increase in success rates for long-horizon, compositional tasks over the baseline on the recently released benchmark data set-ALFRED.

show abstract

Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction

Cited by 13 publications

References 0 publications

Habitat 2.0: Training Home Assistants to Rearrange their Habitat

Habitat 2.0: Training Home Assistants to Rearrange their Habitat

Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scene

A modular vision language navigation and manipulation framework for long horizon compositional tasks in indoor environment

Contact Info

Product

Resources

About