Active Visual Information Gathering for Vision-Language Navigation

Wang, Hanqing; Yang, Yi; Shu, Tianmin; Liang, Wei

doi:10.1007/978-3-030-58542-6_19

Cited by 57 publications

(48 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As a crucial step towards building intelligent robots, autonomous navigation has been long studied in robotics community. Recently vision-based navigation [16] attained growing attention in computer vision community and was explored in many other forms, such as point-goal based or object-oriented (i.e., reaching a specific position or finding an object) [97,16,14,22], naturallanguage-guided [3,86], audio-assisted navigation [17], and from indoor environments [3] to street scenes [18]. Prominent early methods rely on a pre-given/-computed global map [8,44] for path planing, while later ones refer to simultaneous localization and mapping (SLAM) techniques [84,31] that reconstruct map on the fly.…”

Section: Related Workmentioning

confidence: 99%

Collaborative Visual Navigation

Wang,

Zhu

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

As a fundamental problem for Artificial Intelligence, multi-agent system (MAS) is making rapid progress, mainly driven by multi-agent reinforcement learning (MARL) techniques. However, previous MARL methods largely focused on grid-world like or game environments; MAS in visuallyrich environments has remained less explored. To narrow this gap and emphasize the crucial role of perception in MAS, we propose a large-scale 3D dataset, CollaVN, for multi-agent visual navigation (MAVN). In CollaVN, multiple agents are entailed to cooperatively navigate across photo-realistic environments to reach target locations. Diverse MAVN variants are explored to make our problem more general. Moreover, a memory-augmented communication framework is proposed. Each agent is equipped with a private, external memory to persistently store communication information. This allows agents to make better use of their past communication information, enabling more efficient collaboration and robust long-term planning. In our experiments, several baselines and evaluation metrics are designed. We also empirically verify the efficacy of our proposed MARL approach across different MAVN task settings.

show abstract

Section: Related Workmentioning

confidence: 99%

Collaborative Visual Navigation

Wang,

Zhu

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…For further boosting learning, some methods try to mine extra supervisory signals from synthesized samples [13,45,14] or auxiliary tasks [48,25,33,55]. Hence, for more intelligent path planning, the abilities of self-correction [28,34] and active exploration [47] are addressed. Some other recent studies also address environment-agnostic representation learning [49], fine-grained instruction grounding [23,40,22], web imagetext paired data based self-pretraining [35,18], or perform VLN in continuous environments [29].…”

Section: Related Workmentioning

confidence: 99%

“…Since Anderson et al [3] extended prior efforts [9,37] in instruction based navigation into photo-realistic simulated scenes [6], vision-language navigation (VLN) has recently attracted increasing attention in computer vision community. Towards the goal of enabling an agent to execute navigation instructions in 3D environments, current representative VLN methods made great advances in: i) developing more powerful learning paradigms [50,48]; ii) exploring extra supervision signals from synthesized data [13,45,14] or auxiliary tasks [48,25,33,55]; iii) designing more efficient multi-modal embedding schemes [24,40,49]; and iv) making more intelligent path planning [28,34,47]. * Corresponding author: Wenguan Wang.…”

Section: Introductionmentioning

confidence: 99%

Structured Scene Memory for Vision-Language Navigation

Wang¹,

Yang²,

Liang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, numerous algorithms have been developed to tackle the problem of vision-language navigation (VLN), i.e., entailing an agent to navigate 3D environments through following linguistic instructions. However, current VLN agents simply store their past experiences/observations as latent states in recurrent networks, failing to capture environment layouts and make long-term planning. To address these limitations, we propose a crucial architecture, called Structured Scene Memory (SSM). It is compartmentalized enough to accurately memorize the percepts during navigation. It also serves as a structured scene representation, which captures and disentangles visual and geometric cues in the environment. SSM has a collect-read controller that adaptively collects information for supporting current decision making and mimics iterative algorithms for long-range reasoning. As SSM provides a complete action space, i.e., all the navigable places on the map, a frontier-exploration based navigation decision making strategy is introduced to enable efficient and global planning. Experiment results on two VLN datasets (i.e., R2R and R4R) show that our method achieves state-of-the-art performance on several metrics.

show abstract

“…We also analyze the training time and memory consumption at different batch sizes and caption sequence lengths. In addition to image captioning, it should be possible to attain similar benefits in other visual-language tasks such as visual question answering, dialog and vision-language navigation [35][36][37][38][39].…”

Section: Image Captioningmentioning

confidence: 99%

An Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks

2021

View full text Add to dashboard Cite

In the quest to make deep learning systems more capable, a number of more complex, more computationally expensive and memory intensive algorithms have been proposed. This switchover glosses over the capabilities of many of the simpler systems or modules within them to adequately address current and future problems. This has led to some of the deep learning research being inaccessible to researchers who don’t possess top-of-the-line hardware. The use of simple feed forward networks has not been explicitly explored in the current transformer-based vision-language field. In this paper, we use a series of feed-forward layers to encode image features, and caption embeddings, alleviating some of the effects of the computational complexities that accompany the use of the self-attention mechanism and limit its application in long sequence task scenarios. We demonstrate that a decoder does not require masking for conditional short sequence generation where the task is not only dependent on the previously generated sequence, but another input such as image features. We perform an empirical and qualitative analysis of the use of linear transforms in place of self-attention layers in vision-language models, and obtain competitive results on the MSCOCO dataset. Our best feed-forward model obtains average scores of over 90% of the current state-of-the-art pre-trained Oscar model in the conventional image captioning metrics. We also demonstrate that the proposed models take less time training and use less memory at larger batch sizes and longer sequence lengths.

show abstract

Active Visual Information Gathering for Vision-Language Navigation

Cited by 57 publications

References 23 publications

Collaborative Visual Navigation

Collaborative Visual Navigation

Structured Scene Memory for Vision-Language Navigation

An Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks

Contact Info

Product

Resources

About