Vision-Based Navigation With Language-Based Assistance via Imitation Learning With Indirect Intervention

Nguyen, Khanh; Dey, Debadeepta; Brockett, Chris; Dolan, Bill

doi:10.1109/cvpr.2019.01281

Cited by 85 publications

(57 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, we introduce a self-supervised imitation learning method for exploration in order to explicitly address the generalization issue, which is a problem not well-studied in prior work. Concurrent to our work, [44,23,27,28] studies the VLN tasks from various aspects, and [31] introduces a variant of the VLN task to find objects by requesting language assistance when needed. Note that we are the first to propose to explore unseen environments for the VLN task.…”

Section: Related Workmentioning

confidence: 97%

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Wang

Huang

Çelikyılmaz

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

462

387

View full text Add to dashboard Cite

Vision-language navigation (VLN) is the task of navigating an embodied agent to carry out natural language instructions inside real 3D environments. In this paper, we study how to address three critical challenges for this task: the cross-modal grounding, the ill-posed feedback, and the generalization problems. First, we propose a novel Reinforced Cross-Modal Matching (RCM) approach that enforces cross-modal grounding both locally and globally via reinforcement learning (RL). Particularly, a matching critic is used to provide an intrinsic reward to encourage global matching between instructions and trajectories, and a reasoning navigator is employed to perform cross-modal grounding in the local visual scene. Evaluation on a VLN benchmark dataset shows that our RCM model significantly outperforms previous methods by 10% on SPL and achieves the new state-of-the-art performance. To improve the generalizability of the learned policy, we further introduce a Self-Supervised Imitation Learning (SIL) method to explore unseen environments by imitating its own past, good decisions. We demonstrate that SIL can approximate a better and more efficient policy, which tremendously minimizes the success rate performance gap between seen and unseen environments (from 30.7% to 11.7%).

show abstract

Section: Related Workmentioning

confidence: 97%

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Wang

Huang

Çelikyılmaz

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

462

387

View full text Add to dashboard Cite

show abstract

“…Historically, semantic parsing was used to map natural language instructions to visual navigation in simulation environments (Chen and Mooney, 2011;MacMahon et al, 2006). Modern approaches use neural architectures to map natural language to the (simulated) world and execute actions (Paxton et al, 2019;Chen et al, 2018;Nguyen et al, 2018;Blukis et al, 2018;Fried et al, 2018;Mei et al, 2016). In visual question answering (VQA) (Antol et al, 2015;Hudson and Manning, 2019) and visual commonsense reasoning (VCR) (Zellers et al, 2019), input images are accompanied with natural language questions.…”

Section: Related Workmentioning

confidence: 99%

Shifting the Baseline: Single Modality Performance on Visual Navigation &

Thomason¹,

Gordon²,

Bisk³

2019

Proceedings of the 2019 Conference of the North

View full text Add to dashboard Cite

We demonstrate the surprising strength of unimodal baselines in multimodal domains, and make concrete recommendations for best practices in future research. Where existing work often compares against random or majority class baselines, we argue that unimodal approaches better capture and reflect dataset biases and therefore provide an important comparison when assessing the performance of multimodal techniques. We present unimodal ablations on three recent datasets in visual navigation and QA, seeing an up to 29% absolute gain in performance over published baselines.

show abstract

“…Visual-and-language navigation (VLN) [87,88,[118][119][120][121] is a multimodal task that has become increasingly popular in recent years. The idea behind VLN is to combine several active domains (i.e., natural language, vision, and action) to enable robots (intelligent agents) to navigate easily in unstructured environments.…”

Section: Vision-and-language Navigationmentioning

confidence: 99%

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

et al. 2021

View full text Add to dashboard Cite

The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.

show abstract

Vision-Based Navigation With Language-Based Assistance via Imitation Learning With Indirect Intervention

Cited by 85 publications

References 37 publications

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Shifting the Baseline: Single Modality Performance on Visual Navigation &

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

Contact Info

Product

Resources

About