Sim-to-Real Transfer for Vision-and-Language Navigation

Anderson, Peter J.; Shrivastava, Ayush; Truong, Joanne; Majumdar, Arjun; Parikh, Devi; Batra, Dhruv

doi:10.48550/arxiv.2011.03807

Cited by 3 publications

(3 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By using a modular system in conjunction with topological maps, we saw a boost in navigation performance compared to an end-to-end baseline. Potential extensions include demonstrating VLN on a real robot [3] and using topological maps constructed at test time [9]. We hope our work inspires further research in building robot systems which communicate effectively and operate robustly.…”

Section: Discussionmentioning

confidence: 92%

Topological Planning with Transformers for Vision-and-Language Navigation

Chen¹,

Chen²,

Chuang³

et al. 2020

Preprint

View full text Add to dashboard Cite

Conventional approaches to vision-and-language navigation (VLN) are trained end-to-end but struggle to perform well in freely traversable environments. Inspired by the robotics community, we propose a modular approach to VLN using topological maps. Given a natural language instruction and topological map, our approach leverages attention mechanisms to predict a navigation plan in the map. The plan is then executed with low-level actions (e.g. FORWARD, ROTATE) using a robust controller. Experiments show that our method outperforms previous end-to-end approaches, generates interpretable navigation plans, and exhibits intelligent behaviors such as backtracking.

show abstract

Section: Discussionmentioning

confidence: 92%

Topological Planning with Transformers for Vision-and-Language Navigation

Chen¹,

Chen²,

Chuang³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…In another direction, recent approaches have relaxed the constraint of discrete traversal in VLN into continuous space ( [24,25,26,27,28]). Here, the agent needs to deal with higher task complexity involving time and space.…”

Section: Vision-and-language Navigation (Vln) and Robot Navigationmentioning

confidence: 99%

LanguageRefer: Spatial-Language Model for 3D Visual Grounding

Roh¹,

Desingh²,

Farhadi³

et al. 2021

Preprint

View full text Add to dashboard Cite

To realize robots that can understand human instructions and perform meaningful tasks in the near future, it is important to develop learned models that can understand referential language to identify common objects in real-world 3D scenes. In this paper, we develop a spatial-language model for a 3D visual grounding problem. Specifically, given a reconstructed 3D scene in the form of a point cloud with 3D bounding boxes of potential object candidates, and a language utterance referring to a target object in the scene, our model identifies the target object from a set of potential candidates. Our spatial-language model uses a transformer-based architecture that combines spatial embedding from boundingbox with a finetuned language embedding from DistilBert [1] and reasons among the objects in the 3D scene to find the target object. We show that our model performs competitively on visio-linguistic datasets proposed by ReferIt3D [2]. We provide additional analysis of performance in spatial reasoning tasks decoupled from perception noise, the effect of view-dependent utterances in terms of accuracy, and view-point annotations for potential robotics applications. Project website: https://sites.google.com/view/language-refer.

show abstract

“…Recently, [Krantz et al, 2020] breaks through the limitations of the navigation graph and migrate VLN tasks to a continuous environment for the first time. Subsequently, combining a subgoal module with the traditional path planning on the map, [Anderson et al, 2020] transfers the VLN task from the agent in the simulation environment to the physical robot in the real environment and successfully guarantee an acceptable success rate.…”

Section: Introductionmentioning

confidence: 99%

Visual Perception Generalization for Vision-and-Language Navigation via Meta-Learning

Wang¹,

Wu²,

Wang³

2020

Preprint

View full text Add to dashboard Cite

Vision-and-language navigation (VLN) is a challenging task that requires an agent to navigate in real-world environments by understanding natural language instructions and visual information received in real-time. Prior works have implemented VLN tasks on continuous environments or physical robots, all of which use a fixed camera configuration due to the limitations of datasets, such as 1.5 meters height, 90 degrees horizontal field of view (HFOV), etc. However, real-life robots with different purposes have multiple camera configurations, and the huge gap in visual information makes it difficult to directly transfer the learned navigation model between various robots. In this paper, we propose a visual perception generalization strategy based on meta-learning, which enables the agent to fast adapt to a new camera configuration with a few shots. In the training phase, we first locate the generalization problem to the visual perception module, and then compare two meta-learning algorithms for better generalization in seen and unseen environments. One of them uses the Model-Agnostic Meta-Learning (MAML) algorithm that requires a few shot adaptation, and the other refers to a metric-based meta-learning method with a feature-wise affine transformation layer. The experiment results show that our strategy successfully adapts the learned navigation model to a new camera configuration, and the two algorithms show their advantages in seen and unseen environments respectively.

show abstract

Sim-to-Real Transfer for Vision-and-Language Navigation

Cited by 3 publications

References 38 publications

Topological Planning with Transformers for Vision-and-Language Navigation

Topological Planning with Transformers for Vision-and-Language Navigation

LanguageRefer: Spatial-Language Model for 3D Visual Grounding

Visual Perception Generalization for Vision-and-Language Navigation via Meta-Learning

Contact Info

Product

Resources

About