Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigation

Abstract

Vision-and-Language Navigation (VLN) is a task that an agent is required to follow a language instruction to navigate to the goal position, which relies on the ongoing interactions with the environment during moving. Recent Transformer-based VLN methods have made great progress benefiting from the direct connections between visual observations and the language instruction via the multimodal cross-attention mechanism. However, these methods usually represent temporal context as a fixed-length vector by using an LSTM decoder or using manually designed hidden states to build a recurrent Transformer. Considering a single fixed-length vector is often insufficient to capture longterm temporal context, in this paper, we introduce Multimodal Transformer with Variable-length Memory (MTVM) for visually-grounded natural language navigation by modelling the temporal context explicitly. Specifically, MTVM enables the agent to keep track of the navigation trajectory by directly storing previous activations in a memory bank. To further boost the performance, we propose a memoryaware consistency loss to help learn a better joint representation of temporal context with random masked instructions. We evaluate MTVM on popular R2R and CVDN datasets, and our model improves Success Rate on R2R unseen validation and test set by 2% each, and reduce Goal Process by 1.6m on CVDN test set.* This work was performed while Chuang Lin worked as an intern at ByteDance.

Incorporating External Knowledge Reasoning for Vision-and-Language Navigation with Assistant’s Help

“…Recently, HAMT [42] and Episodic Transformer [43] explicitly modeled the history information by directly encoding all past observations and actions, but this is fairly complex. In contrast, MTVM [5] proposes a Transformer with a variable-length memory framework to model history information explicitly by copying past activations into a memory bank without the need to consider distances in the path.…”

“…Remembering history information is essential for an agent to implement correct decision-making during navigation. Inspired by [5], we introduce a memory-based Trans-former module as the action policy framework to explicitly model history information. The information generated during reasoning is stored in the form of scene memory token m t , where m t is composed of multiple features concatenated:…”

“…In this paper, we propose to combine the power of Transformer and external knowledge to improve the agent's performance in VNLA. Firstly, inspired by [5], we design a memory-based Transformer with variable-length input as the policy module for VNLA, where the visual and linguistic clues constitute a scene memory token m t at each time step t. The token m t is stored in the memory bank M ordered by the time sequence, which allows explicitly modeling history information, and the Transformer architecture naturally accommodates variable-length memory inputs to predict the next action. The information clues are effectively extracted from history tokens for the current decision-making of the agent through a learnable multi-layer attention.…”

Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend

“…Most neural network models cannot read and write long-term memory parts and cannot be tightly coupled with inference. The method based on a memory network stores the extracted features in external memory and designs a pairing and reading algorithm, which has a good effect on improving the modal information inference of sequences and is effective on video captioning [21], vision-and-language navigation [22,23], and visual-and-textual question answering [24] tasks.…”

Target-Driven Structured Transformer Planner for Vision-Language Navigation

“…In addition, as the VLN agent faces structured environments, understanding the topology of the environment is crucial for the success of navigation. However, existing methods either arrange the historical observations in a sequential manner [14,32,42], or adopt complicated modules (e.g., graph neural networks) for modeling environment layouts [10,22,53]. Differently, we develop a Structured Transformer Planner (STP), where the position-embedded visual observations are used as input tokens, and the geometric relations (local connectivity) among the navigation locations are elegantly formulated as the directional attention among input tokens.…”