Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation

Abstract

In reinforcement learning for visual navigation, it is common to develop a model for each new task, and train that model from scratch with task-specific interactions in 3D environments. However, this process is expensive; massive amounts of interactions are needed for the model to generalize well. Moreover, this process is repeated whenever there is a change in the task type or the goal modality. We present a unified approach to visual navigation using a novel modular transfer learning model. Our model can effectively leverage its experience from one source task and apply it to multiple target tasks (e.g., ObjectNav, RoomNav, ViewNav) with various goal modalities (e.g., image, sketch, audio, label). Furthermore, our model enables zero-shot experience learning, whereby it can solve the target tasks without receiving any task-specific interactive training. Our experiments on multiple photorealistic datasets and challenging tasks show that our approach learns faster, generalizes better, and outperforms SoTA models by a significant margin.

ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings

“…During pre-processing, we further augment the dataset by sampling goal-images at four evenly-spaced heading angles to produce 36M total episodes for training. Sampling at multiple angles approximates the randomized sampling used in [18].…”

“…We perform large-scale experiments on three ObjectNav datasets -Gibson [4], MP3D [8], and HM3D [20]. Our zero-shot agent (that has not seen a single 3D semantic annotation or ObjectNav training episode) achieves a 31.3% success in Gibson environments, which is a 20.0% absolute improvement over previous zero-shot results [18]. In MP3D, our agent achieves 15.3% success, a 4.2% absolute gain over existing zero-shot methods [21].…”

“…Zero-Shot ObjectNav. Two recent works [18,21] directly address our motivation (zero-shot ObjectNav) and are most related. First, ZER [18] proposes a two-stage framework in which an image-goal navigation (ImageNav) agent is first trained from scratch.…”

“…Two recent works [18,21] directly address our motivation (zero-shot ObjectNav) and are most related. First, ZER [18] proposes a two-stage framework in which an image-goal navigation (ImageNav) agent is first trained from scratch. Then, independent encoders are trained to map from various modalities (including language) into the image-goal embedding space.…”

“…During RL training, we use two data augmentation techniques: color jitter and random translation (adapted from [16]). Specifically, we train with DD-PPO [32] using a reward function proposed for ImageNav by Al-Halah et al [18]:…”