The Regretful Agent: Heuristic-Aided Navigation Through Progress Estimation

Abstract

I know I came from there. Where should I go next? My estimated confidence decreased. Something went wrong. Let's learn this lesson and go back. Instruction: Exit the room. Walk past the display case and into the kitchen. Stop by the table. 20% 13% 25% 42% 60% 75% 90% 1 st step 1 st step 2 nd 5 th 5 th step 4 th 6 th 7 th Figure 1: Vision-and-Language Navigation task and our proposed regretful navigation agent. The agent leverages the selfmonitoring mechanism [14] through time to decide when to roll back to a previous location and resume the instructionfollowing task. Our code is available at https://github.com/chihyaoma/regretful-agent. AbstractAs deep learning continues to make progress for challenging perception tasks, there is increased interest in combining vision, language, and decision-making. Specifically, the Vision and Language Navigation (VLN) task involves navigating to a goal purely from language instructions and visual information without explicit knowledge of the goal. Recent successful approaches have made in-roads in achieving good success rates for this task but rely on beam search, which thoroughly explores a large number of trajectories and is unrealistic for applications such as robotics. In this paper, inspired by the intuition of viewing the problem as search on a navigation graph, we propose to use a progress monitor developed in prior work as a learnable heuristic for search. We then propose two modules incorporated into an end-to-end architecture: 1) A learned mechanism to perform backtracking, which decides whether to continue moving forward or roll back to a previous state (Regret Module) and 2) A mechanism to help the agent decide which direction to go next by showing directions that are visited and their associated progress estimate (Progress * Work partially done while the author was a research intern at Salesforce Research.Marker). Combined, the proposed approach significantly outperforms current state-of-the-art methods using greedy action selection, with 5% absolute improvement on the test server in success rates, and more importantly 8% on success rates normalized by the path length.

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

“…Moreover, we introduce a self-supervised imitation learning method for exploration in order to explicitly address the generalization issue, which is a problem not well-studied in prior work. Concurrent to our work, [44,23,27,28] studies the VLN tasks from various aspects, and [31] introduces a variant of the VLN task to find objects by requesting language assistance when needed. Note that we are the first to propose to explore unseen environments for the VLN task.…”

Robust Navigation with Language Pretraining and Stochastic Sampling

“…2 To compensate, beam search is often used to improve success rates. Recent work, e.g., using search strategies (Ke et al, 2019) or progress monitors (Ma et al, 2019b), has focused on mitigating the cost of computing top-k rollouts.…”

“…• RCM+SIL(TRAIN) (Wang et al, 2019): an agent trained with cross-modal grounding locally and globally via reinforcement learning. • REGRETFUL (Ma et al, 2019b): an agent with a trained progress monitor heuristic for search that enables backtracking. • FAST (Ke et al, 2019): an agent which combines global and local knowledge to compare partial trajectories of different lengths, enabling efficient backtrack after a mistake.…”

Proceedings of the Combined Workshop on Spatial Language Understanding (

“…The dataset presents a unique challenge as it not only substitutes virtual environments (e.g., MacMahon et al (2006)) with photo-realistic environments but also describes the paths in the environment using human-annotated instructions (as opposed to formulaic instructions provided by mapping applications e.g., ). A number of methods (Anderson et al, 2018b;Fried et al, 2018;Wang et al, 2018a;Ma et al, 2019a;Wang et al, 2018b;Ma et al, 2019b) have been proposed recently to solve the navigation task described in R2R dataset. All these methods build models for agents that learn to navigate in R2R environment and are trained on the entire R2R dataset as well as the augmented dataset introduced by Fried et al 2018which is generated by a speaker model trained on human-annotated instructions.…”