Vision-Language Navigation with Random Environmental Mixup

Abstract

Vision-language Navigation (VLN) tasks require an agent to navigate step-by-step while perceiving the visual observations and comprehending a natural language instruction. Large data bias, which is caused by the disparity ratio between the small data scale and large navigation space, makes the VLN task challenging. Previous works have proposed various data augmentation methods to reduce data bias. However, these works do not explicitly reduce the data bias across different house scenes. Therefore, the agent would overfit to the seen scenes and achieve poor navigation performance in the unseen scenes. To tackle this problem, we propose the Random Environmental Mixup (REM) method, which generates cross-connected house scenes as augmented data via mixuping environment. Specifically, we first select key viewpoints according to the room connection graph for each scene. Then, we crossconnect the key views of different scenes to construct augmented scenes. Finally, we generate augmented instructionpath pairs in the cross-connected scenes. The experimental results on benchmark datasets demonstrate that our augmentation data via REM help the agent reduce its performance gap between the seen and unseen environment and improve the overall performance, making our model the best existing approach on the standard VLN benchmark.

Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigation

“…[36] proposes an environmental dropout method based on the view consistency to mimic novel and diverse environments. From a different perspective, REM [24] reconnect the seen scenes to generate augmented data via mixing up environments. To further understand the relations between the instructions and scenes, [16] and [33] take the objects in scenes and the corresponding words in instructions as the minimal units of encoding.…”

“…However, its end-to-end training requires 20 NVIDIA V100 GPUs for ∼ 20 hours, which is much higher than ours (3 V100 for 1 day). Another recent work Mixup [24] extends 61 scenes in the training set to 116 cross-connected scenes with data augmentation and thus uses more data than other methods. We leave more comparisons in Appendix.…”

“…In the past few years, many methods [3,10,16,20,23,24,27,31,36,40,41,44] have been proposed for the VLN task. Most of them adopt the encoder-decoder framework which first encodes the instruction and visual observations and then decodes the action sequence.…”

Explore the Potential Performance of Vision-and-Language Navigation Model: a Snapshot Ensemble Method

“…We apply our modification and snapshot ensemble on the VLN BERT model proposed by Hong et al (2021). 3 The model currently holds the best performance for the singlerun setting in the R2R dataset (Liu et al 2021). In this section, we will have a brief recap of this model.…”

“…Previous studies have also made efforts to prevent overfitting due to the limited size of the R2R dataset (Fried et al 2018;Liu et al 2021;Li et al 2019;Hao et al 2020).…”