Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigation

Narasimhan, Medhini; Wijmans, Erik; Chen, Xinlei; Darrell, Trevor; Batra, Dhruv; Parikh, Devi; Singh, Amanpreet

doi:10.48550/arxiv.2007.09841

Cited by 10 publications

(14 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Liang et al [41] design a semantic scene completion module to complete the unexplored scene. Similar to it, some methods achieve image-level extrapolation of depth and semantics [45], high-dimension feature space extrapolation [46], semantic scene completion [42], [43], attention probability modeling [44], and room-type prediction [47]. We formulate the ObjectNav task as target distance prediction and path planning.…”

Section: B Learning-based Goal Navigation Methodsmentioning

confidence: 99%

Navigating to Objects in Unseen Environments by Distance Prediction

Zhu¹,

Zhao²,

Kong³

2022

Preprint

View full text Add to dashboard Cite

Object Goal Navigation (ObjectNav) task is to navigate an agent to an object instance in unseen environments. The traditional navigation paradigm plans the shortest path on a pre-built map. Inspired by this, we propose an object goal navigation framework, which could directly perform path planning based on an estimated distance map. Specifically, our model takes a birds-eye-view semantic map as input, and estimates the distance from the map cells to the target object based on the learned prior knowledge. With the estimated distance map, the agent could explore the environment and navigate to the target objects based on either human-designed or learned navigation policy. Empirical results in visually realistic simulation environments show that the proposed method outperforms a wide range of baselines on success rate and efficiency.

show abstract

Section: B Learning-based Goal Navigation Methodsmentioning

confidence: 99%

Navigating to Objects in Unseen Environments by Distance Prediction

Zhu¹,

Zhao²,

Kong³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…[55] uses Graph Convolutional Networks to incorporate prior semantic knowledge in a deep reinforcement learning framework, while [14] learns semantic associations by defining a topological map over the 2D scene. Finally, [38] learns to predict room layouts in unobserved regions of the map in an attempt to model architectural regularities in houses for the task of room navigation. In contrast to all these works we formulate an active, target-independent strategy to predict semantic maps and define goal selection objectives.…”

Section: Related Workmentioning

confidence: 99%

“…Moreover, the semantic priors are usually encoded implicitly by goal oriented navigation policy functions [14] and are, thus, target-dependent. More relevant to ours, other works have introduced spatial prediction models that either anticipate occupancy [42] or room layouts [38] beyond the agent's field of view and demonstrated improved performance on navigation tasks. Our work differs from these methods in three principled ways: 1) We formulate an active training strategy for learning the semantic maps, 2) we exploit the uncertainty over the predictions in the planning process, and 3) while occupancy anticipation mainly learns to extend what it already sees, applying the same concept to semantics is not trivial.…”

Section: Introductionmentioning

confidence: 96%

Learning to Map for Active Semantic Goal Navigation

Georgakis¹,

Bucher²,

Schmeckpeper³

et al. 2021

Preprint

View full text Add to dashboard Cite

We consider the problem of object goal navigation in unseen environments. In our view, solving this problem requires learning of contextual semantic priors, a challenging endeavour given the spatial and semantic variability of indoor environments. Current methods learn to implicitly encode these priors through goal-oriented navigation policy functions operating on spatial representations that are limited to the agent's observable areas. In this work, we propose a novel framework that actively learns to generate semantic maps outside the field of view of the agent and leverages the uncertainty over the semantic classes in the unobserved areas to decide on long term goals. We demonstrate that through this spatial prediction strategy, we are able to learn semantic priors in scenes that can be leveraged in unknown environments. Additionally, we show how different objectives can be defined by balancing exploration with exploitation during searching for semantic targets. Our method is validated in the visually realistic environments offered by the Matterport3D dataset and show state of the art results on the object goal navigation task.* Denotes equal contribution.

show abstract

“…with as few steps as possible). The internal model can be in forms like a topological graph map [51], semantic map [52], occupancy map [53] or spatial memory [54,55]. These map-based architectures can capture geometry and semantics, allowing for more efficient policy learning and planning [53] as compared to reactive and recurrent neural network policies [56].…”

Section: Visual Explorationmentioning

confidence: 99%

A Survey of Embodied AI: From Simulators to Research Tasks

Duan¹,

Jian²,

Tan³

et al. 2021

Preprint

View full text Add to dashboard Cite

There has been an emerging paradigm shift from the era of "internet AI" to "embodied AI", whereby AI algorithms and agents no longer simply learn from datasets of images, videos or text curated primarily from the internet. Instead, they learn through embodied physical interactions with their environments, whether real or simulated. Consequently, there has been substantial growth in the demand for embodied AI simulators to support a diversity of embodied AI research tasks. This growing interest in embodied AI is beneficial to the greater pursuit of artificial general intelligence, but there is no contemporary and comprehensive survey of this field. This paper comprehensively surveys state-of-the-art embodied AI simulators and research, mapping connections between these. By benchmarking nine state-of-the-art embodied AI simulators in terms of seven features, this paper aims to understand the simulators in their provision for use in embodied AI research. Finally, based upon the simulators and a pyramidal hierarchy of embodied AI research tasks, this paper surveys the main research tasks in embodied AIvisual exploration, visual navigation and embodied question answering (QA), covering the state-of-the-art approaches, evaluation and datasets.

show abstract

Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigation

Cited by 10 publications

References 43 publications

Navigating to Objects in Unseen Environments by Distance Prediction

Navigating to Objects in Unseen Environments by Distance Prediction

Learning to Map for Active Semantic Goal Navigation

A Survey of Embodied AI: From Simulators to Research Tasks

Contact Info

Product

Resources

About