Memory-Augmented Reinforcement Learning for Image-Goal Navigation

Abstract

In this work, we address the problem of image-goal navigation in the context of visually-realistic 3D environments. This task involves navigating to a location indicated by a target image in a previously unseen environment. Earlier attempts, including RL-based and SLAM-based approaches, have either shown poor generalization performance, or are heavily-reliant on pose/depth sensors. We present a novel method that leverages a cross-episode memory to learn to navigate. We first train a state-embedding network in a self-supervised fashion, and then use it to embed previously-visited states into an agent's memory. In order to avoid overfitting, we propose to use data augmentation on the RGB input during training. We validate our approach through extensive evaluations, showing that our data-augmented memory-based model establishes a new state of the art on the image-goal navigation task in the challenging Gibson dataset. We obtain this competitive performance from RGB input only, without access to additional sensors such as position or depth.

No RL, No Simulation: Learning to Navigate without Navigating

“…With increased data and steps RL baselines unsurprisingly increase in performance, however we find with 5X more data and 10x more compute RL baselines still are outperformed by NRNS. The low performance of behavioral cloning and RL methods for image-goal navigation is unsurprising [4,15]. This demonstrates the difficulty of learning rewards on low level actions instead of value learning on possible exploration directions, exacerbating the difficulty of exploration in image-goal navigation.…”

“…Navigation tasks largely fall into two main categories [1], ones in which a goal location is known [11,12,13] and limited exploration is required, and ones in which the goal location is not known and efficient exploration is necessary. In the second category, tasks range from finding the location of specific objects [5], rooms [14], or images [15], to the task of exploration itself [2]. The majority of current work [12,15,16,3] leverages simulators [7] and extensive interaction to learn end-to-end models for these tasks.…”

“…In the second category, tasks range from finding the location of specific objects [5], rooms [14], or images [15], to the task of exploration itself [2]. The majority of current work [12,15,16,3] leverages simulators [7] and extensive interaction to learn end-to-end models for these tasks. In contrast, our work shows that the semantic cues needed for exploration-based navigation tasks can be learned directly from video trajectories.…”

Active Audio-Visual Separation of Dynamic Sound Sources

“…Transformer Memory. Transformers [71] have been shown to do very well on long-horizon embodied tasks like navigation and exploration [11,13,17,24,46,47,57]. The performance gains arise from a transformer's ability to effectively leverage past experiences of the agent [11,24,46,47,57] and also to do cross-modal reasoning [13,17].…”

“…Transformers [71] have been shown to do very well on long-horizon embodied tasks like navigation and exploration [11,13,17,24,46,47,57]. The performance gains arise from a transformer's ability to effectively leverage past experiences of the agent [11,24,46,47,57] and also to do cross-modal reasoning [13,17]. Different from these methods, our idea is to use a transformer as a memory model for capturing long-range acoustic correlations for audio-visual separation.…”