Auxiliary Tasks and Exploration Enable ObjectGoal Navigation

Ye, Joel; Batra, Dhruv; Das, Abhishek; Wijmans, Erik

doi:10.1109/iccv48922.2021.01581

Cited by 87 publications

(66 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In our experiments, we use offline datasets of size 20K images or less in which the "annotations" are actually automatic object detections. This is several orders of magnitude smaller than the amount of interactions usually needed to train a targetspecific policy (tens to hundreds of millions) [45,69].…”

Section: Joint Goal Embedding Learningmentioning

confidence: 97%

Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation

Al-Halah¹,

Ramakrishnan²,

Grauman³

2022

Preprint

View full text Add to dashboard Cite

In reinforcement learning for visual navigation, it is common to develop a model for each new task, and train that model from scratch with task-specific interactions in 3D environments. However, this process is expensive; massive amounts of interactions are needed for the model to generalize well. Moreover, this process is repeated whenever there is a change in the task type or the goal modality. We present a unified approach to visual navigation using a novel modular transfer learning model. Our model can effectively leverage its experience from one source task and apply it to multiple target tasks (e.g., ObjectNav, RoomNav, ViewNav) with various goal modalities (e.g., image, sketch, audio, label). Furthermore, our model enables zero-shot experience learning, whereby it can solve the target tasks without receiving any task-specific interactive training. Our experiments on multiple photorealistic datasets and challenging tasks show that our approach learns faster, generalizes better, and outperforms SoTA models by a significant margin.

show abstract

Section: Joint Goal Embedding Learningmentioning

confidence: 97%

Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation

Al-Halah¹,

Ramakrishnan²,

Grauman³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Termination criterion. We aim for complete map construction, however, in the iGibson environment, all the robots have physical bodies and get stuck occasionally [47]. Hence following [16], our algorithm terminates when there is no accessible frontier in the environment.…”

Section: Methodsmentioning

confidence: 99%

Multi-Robot Active Mapping via Neural Bipartite Graph Matching

Yang¹,

Dong²,

Fan³

et al. 2022

Preprint

View full text Add to dashboard Cite

We study the problem of multi-robot active mapping, which aims for complete scene map construction in minimum time steps. The key to this problem lies in the goal position estimation to enable more efficient robot movements. Previous approaches either choose the frontier as the goal position via a myopic solution that hinders the time efficiency, or maximize the long-term value via reinforcement learning to directly regress the goal position, but does not guarantee the complete map construction. In this paper, we propose a novel algorithm, namely NeuralCoMapping, which takes advantage of both approaches. We reduce the problem to bipartite graph matching, which establishes the node correspondences between two graphs, denoting robots and frontiers. We introduce a multiplex graph neural network (mGNN) that learns the neural distance to fill the affinity matrix for more effective graph matching. We optimize the mGNN with a differentiable linear assignment layer by maximizing the long-term values that favor time efficiency and map completeness via reinforcement learning. We compare our algorithm with several state-of-the-art multi-robot active mapping approaches and adapted reinforcement-learning baselines. Experimental results demonstrate the superior performance and exceptional generalization ability of our algorithm on various indoor scenes and unseen number of robots, when only trained with 9 indoor scenes.

show abstract

“…Perhaps most related to our work is the work of Ye [46], who show that using auxiliary tasks can improve PointGoal navigation results in the Gibson environment [44]. Differ-…”

Section: Related Workmentioning

confidence: 95%

Curious Representation Learning for Embodied Intelligence

Du¹,

Gan²,

Isola³

2021

Preprint

View full text Add to dashboard Cite

Self-supervised representation learning has achieved remarkable success in recent years. By subverting the need for supervised labels, such approaches are able to utilize the numerous unlabeled images that exist on the Internet and in photographic datasets. Yet to build truly intelligent agents, we must construct representation learning algorithms that can learn not only from datasets but also learn from environments. An agent in a natural environment will not typically be fed curated data. Instead, it must explore its environment to acquire the data it will learn from. We propose a framework, curious representation learning (CRL), which jointly learns a reinforcement learning policy and a visual representation model. The policy is trained to maximize the error of the representation learner, and in doing so is incentivized to explore its environment. At the same time, the learned representation becomes stronger and stronger as the policy feeds it ever harder data to learn from. Our learned representations enable promising transfer to downstream navigation tasks, performing better than or comparably to ImageNet pretraining without using any supervision at all. In addition, despite being trained in simulation, our learned representations can obtain interpretable results on real images.

show abstract

Auxiliary Tasks and Exploration Enable ObjectGoal Navigation

Cited by 87 publications

References 13 publications

Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation

Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation

Multi-Robot Active Mapping via Neural Bipartite Graph Matching

Curious Representation Learning for Embodied Intelligence

Contact Info

Product

Resources

About