Diagnosing the Environment Bias in Vision-and-Language Navigation

Abstract

<jats:p>Vision-and-Language Navigation (VLN) requires an agent to follow natural-language instructions, explore the given environments, and reach the desired target locations. These step-by-step navigational instructions are crucial when the agent is navigating new environments about which it has no prior knowledge. Most recent works that study VLN observe a significant performance drop when tested on unseen environments (i.e., environments not used in training), indicating that the neural agent models are highly biased towards training environments. Although this issue is considered as one of the major challenges in VLN research, it is still under-studied and needs a clearer explanation. In this work, we design novel diagnosis experiments via environment re-splitting and feature replacement, looking into possible reasons for this environment bias. We observe that neither the language nor the underlying navigational graph, but the low-level visual appearance conveyed by ResNet features directly affects the agent model and contributes to this environment bias in results. According to this observation, we explore several kinds of semantic representations that contain less low-level visual information, hence the agent learned with these features could be better generalized to unseen testing environments. Without modifying the baseline agent model and its training method, our explored semantic features significantly decrease the performance gaps between seen and unseen on multiple datasets (i.e. R2R, R4R, and CVDN) and achieve competitive unseen results to previous state-of-the-art models.</jats:p>

L2C: Describing Visual Differences Needs Semantic Understanding of Individuals

“…Therefore, we believe that visual differences should be learned by understanding and comparing every single image's semantic representation. A most recent work (Zhang et al, 2020) conceptually supports this argument, where they show that low-level ResNet visual features lead to poor generalization in vision-and-language navigation, and high-level semantic segmentation helps the agent Image I 1…”

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

“…Visual-and-language navigation (VLN) [87,88,[118][119][120][121] is a multimodal task that has become increasingly popular in recent years. The idea behind VLN is to combine several active domains (i.e., natural language, vision, and action) to enable robots (intelligent agents) to navigate easily in unstructured environments.…”

Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring

“…Many models have been developed, such as the Speaker-Follower model (Fried et al, 2018), the Self-Monitoring Navigation Agent (Ma et al, 2019a;Ke et al, 2019), the Regretful Agent (Ma et al, 2019b), and the environment drop-out model (Tan et al, 2019). The VLN benchmark is further extended to study the fidelity of instruction following (Jain et al, 2019) and examined to understand the bias of the benchmark (Zhang et al, 2020). Beyond navigation, there are also benchmarks that additionally incorporate object manipulation to broaden research on vision and language reasoning, such as embodied question answering (Das et al, 2018a;Gordon et al, 2018).…”