Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigation

Narasimhan, Medhini; Wijmans, Erik; Chen, Xinlei; Darrell, Trevor; Batra, Dhruv; Parikh, Devi; Singh, Amanpreet

doi:10.1007/978-3-030-58523-5_30

Cited by 41 publications

(19 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…a room (e.g., go to the kitchen) [50,56,65]. In AudioNav, the agent must find a sounding target (e.g., find the ringing phone) [17,29].…”

Section: Introductionmentioning

confidence: 99%

Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation

Al-Halah¹,

Ramakrishnan²,

Grauman³

2022

Preprint

View full text Add to dashboard Cite

In reinforcement learning for visual navigation, it is common to develop a model for each new task, and train that model from scratch with task-specific interactions in 3D environments. However, this process is expensive; massive amounts of interactions are needed for the model to generalize well. Moreover, this process is repeated whenever there is a change in the task type or the goal modality. We present a unified approach to visual navigation using a novel modular transfer learning model. Our model can effectively leverage its experience from one source task and apply it to multiple target tasks (e.g., ObjectNav, RoomNav, ViewNav) with various goal modalities (e.g., image, sketch, audio, label). Furthermore, our model enables zero-shot experience learning, whereby it can solve the target tasks without receiving any task-specific interactive training. Our experiments on multiple photorealistic datasets and challenging tasks show that our approach learns faster, generalizes better, and outperforms SoTA models by a significant margin.

show abstract

“…a room (e.g., go to the kitchen) [50,56,65]. In AudioNav, the agent must find a sounding target (e.g., find the ringing phone) [17,29].…”

Section: Introductionmentioning

confidence: 99%

Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation

Al-Halah¹,

Ramakrishnan²,

Grauman³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Recent methods focus on learning policies for efficient exploration either through coverage [9], [13], [36], [37] or map accuracy [12] reward functions. Furthermore, several works have gone beyond traditional mapping, and sought to predict maps for unseen regions [12], [38], [24], [27], [39] which further increased robustness in the decision making process. Our approach leverages the uncertainty over predicted occupancy maps for unobserved areas and shows its effectiveness on exploring a novel environment.…”

Section: Related Workmentioning

confidence: 99%

Uncertainty-driven Planner for Exploration and Navigation

Georgakis¹,

Bucher²,

Arapin³

et al. 2022

Preprint

View full text Add to dashboard Cite

We consider the problems of exploration and point-goal navigation in previously unseen environments, where the spatial complexity of indoor scenes and partial observability constitute these tasks challenging. We argue that learning occupancy priors over indoor maps provides significant advantages towards addressing these problems. To this end, we present a novel planning framework that first learns to generate occupancy maps beyond the field-of-view of the agent, and second leverages the model uncertainty over the generated areas to formulate path selection policies for each task of interest. For point-goal navigation the policy chooses paths with an upper confidence bound policy for efficient and traversable paths, while for exploration the policy maximizes model uncertainty over candidate paths. We perform experiments in the visually realistic environments of Matterport3D using the Habitat simulator and demonstrate: 1) Improved results on exploration and map quality metrics over competitive methods, and 2) The effectiveness of our planning module when paired with the stateof-the-art DD-PPO method for the point-goal navigation task.

show abstract

“…The community has built several indoor navigation simulators [41,57,40,27] on top of photo-realistic scans of 3D environments [27,6,47,56,55]. To test a robot's ability to perceive, navigate and interact with the environment, the community has also introduced several tasks [57,5,45,10,52,36,3,28,48,22,21,51,16,34,33,31,32] and benchmarks. Specifically, Batra et al [5] introduce evaluation details for the task of Object Navigation, requiring the agent to navigate to a given object class instead of a final point-goal.…”

Section: Related Workmentioning

confidence: 99%

“…Specifically, Batra et al [5] introduce evaluation details for the task of Object Navigation, requiring the agent to navigate to a given object class instead of a final point-goal. Similarly, Room Navigation [36] requires the agent to navigate to a given room type. More recently, Krantz et al [45,28,48] extend the navigation task to utilize instructions in natural language.…”

Section: Related Workmentioning

confidence: 99%

The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation

Zhao¹,

Agrawal

Batra

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

It is fundamental for personal robots to reliably navigate to a specified goal. To study this task, PointGoal navigation has been introduced in simulated Embodied AI environments. Recent advances solve this PointGoal navigation task with near-perfect accuracy (99.6% success) in photo-realistically simulated environments, assuming noiseless egocentric vision, noiseless actuation and most importantly, perfect localization. However, under realistic noise models for visual sensors and actuation, and without access to a "GPS and Compass sensor," the 99.6%-success agents for PointGoal navigation only succeed with 0.3%. 1 In this work, we demonstrate the surprising effectiveness of visual odometry for the task of PointGoal navigation in this realistic setting, i.e., with realistic noise models for perception and actuation and without access to GPS and Compass sensors. We show that integrating visual odometry techniques into navigation policies improves the state-of-the-art on the popular Habitat PointNav benchmark by a large margin, improving success from 64.5% to 71.7% while executing 6.4 times faster.

show abstract

Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigation

Cited by 41 publications

References 31 publications

Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation

Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation

Uncertainty-driven Planner for Exploration and Navigation

The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation

Contact Info

Product

Resources

About