Learning to Visually Navigate in Photorealistic Environments Without any Supervision

Mezghani, Lina; Sukhbaatar, Sainbayar; Szlam, Arthur; Joulin, Armand; Bojanowski, Piotr

doi:10.48550/arxiv.2004.04954

Cited by 6 publications

(15 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…limited number of steps) before the start of navigation [17]. In the latter case, the agent builds the map as it navigates an unseen test environment [57,58,44], which makes it more tightly integrated with the downstream task. In this section, we build upon existing visual exploration survey papers [48,47] to include more recent works and directions.…”

Section: Visual Explorationmentioning

confidence: 99%

See 1 more Smart Citation

A Survey of Embodied AI: From Simulators to Research Tasks

Duan¹,

Jian²,

Tan³

et al. 2021

Preprint

View full text Add to dashboard Cite

There has been an emerging paradigm shift from the era of "internet AI" to "embodied AI", whereby AI algorithms and agents no longer simply learn from datasets of images, videos or text curated primarily from the internet. Instead, they learn through embodied physical interactions with their environments, whether real or simulated. Consequently, there has been substantial growth in the demand for embodied AI simulators to support a diversity of embodied AI research tasks. This growing interest in embodied AI is beneficial to the greater pursuit of artificial general intelligence, but there is no contemporary and comprehensive survey of this field. This paper comprehensively surveys state-of-the-art embodied AI simulators and research, mapping connections between these. By benchmarking nine state-of-the-art embodied AI simulators in terms of seven features, this paper aims to understand the simulators in their provision for use in embodied AI research. Finally, based upon the simulators and a pyramidal hierarchy of embodied AI research tasks, this paper surveys the main research tasks in embodied AIvisual exploration, visual navigation and embodied question answering (QA), covering the state-of-the-art approaches, evaluation and datasets.

show abstract

Section: Visual Explorationmentioning

confidence: 99%

“…Examples of downstream tasks that make use of visual exploration outputs (i.e. maps) include Image Navigation [51,57], Point Navigation [45,17] and Object Navigation [75,76,77]. More details about these navigation tasks can be found in Section 4.3.1…”

Section: Evaluation Metricsmentioning

confidence: 99%

A Survey of Embodied AI: From Simulators to Research Tasks

Duan¹,

Jian²,

Tan³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Since deep learning methods have revealed their ability in feature engineering, end-to-end agents are becoming popular. Later works [16,51,33] adopt the idea of SLAM and introduce a memory mechanism, a method combining classical mapping methods and deep learning methods for generalization and long-trajectory navigation purposes. Recent works [9,14,8] model the navigation semantics in graphs and achieve great success in embodied navigation tasks.…”

Section: Related Workmentioning

confidence: 99%

“…Recent research efforts [49,19,17,47,33,45,32] have achieved great success in embodied navigation tasks. The agent is able to reach the target by following a variety of instructions, such as a word (e.g.…”

Section: Introductionmentioning

confidence: 99%

SOON: Scenario Oriented Object Navigation with Graph-based Exploration

Zhu¹,

Liang²,

Zhu³

et al. 2021

Preprint

View full text Add to dashboard Cite

The ability to navigate like a human towards a languageguided target from anywhere in a 3D embodied environment is one of the 'holy grail' goals of intelligent robots. Most visual navigation benchmarks, however, focus on navigating toward a target from a fixed starting point, guided by an elaborate set of instructions that depicts step-by-step. This approach deviates from real-world problems in which human-only describes what the object and its surrounding look like and asks the robot to start navigation from anywhere. Accordingly, in this paper, we introduce a Scenario Oriented Object Navigation (SOON) task. In this task, an agent is required to navigate from an arbitrary position in a 3D embodied environment to localize a target following a scene description. To give a promising direction to solve this task, we propose a novel graph-based exploration (GBE) method, which models the navigation state as a graph and introduces a novel graph-based exploration approach to learn knowledge from the graph and stabilize training by learning sub-optimal trajectories. We also propose a new large-scale benchmark named From Anywhere to Object (FAO) dataset. To avoid target ambiguity, the descriptions in FAO provide rich semantic scene information includes: object attribute, object relationship, region description, and nearby region description. Our experiments reveal that the proposed GBE outperforms various state-of-the-arts on both FAO and R2R datasets. And the ablation studies on FAO validates the quality of the dataset.

show abstract

“…Previous work has either shown results in a limited setting with synthetic data [15] or reported poor RL-based performance [9]. Following Chaplot et al citechaplot2020neural, and in contrast to some previous works [22,37], we aim to move away from such limited setups and target unseen environments. This requires generalization from the agent's policy, for which there are no guarantees or known cooking recipes.…”

Section: Introductionmentioning

confidence: 99%

Memory-Augmented Reinforcement Learning for Image-Goal Navigation

Mezghani,

Sukhbaatar,

Lavril

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In this work, we address the problem of image-goal navigation in the context of visually-realistic 3D environments. This task involves navigating to a location indicated by a target image in a previously unseen environment. Earlier attempts, including RL-based and SLAM-based approaches, have either shown poor generalization performance, or are heavily-reliant on pose/depth sensors. We present a novel method that leverages a cross-episode memory to learn to navigate. We first train a state-embedding network in a self-supervised fashion, and then use it to embed previously-visited states into an agent's memory. In order to avoid overfitting, we propose to use data augmentation on the RGB input during training. We validate our approach through extensive evaluations, showing that our data-augmented memory-based model establishes a new state of the art on the image-goal navigation task in the challenging Gibson dataset. We obtain this competitive performance from RGB input only, without access to additional sensors such as position or depth.

show abstract

Learning to Visually Navigate in Photorealistic Environments Without any Supervision

Cited by 6 publications

References 0 publications

A Survey of Embodied AI: From Simulators to Research Tasks

A Survey of Embodied AI: From Simulators to Research Tasks

SOON: Scenario Oriented Object Navigation with Graph-based Exploration

Memory-Augmented Reinforcement Learning for Image-Goal Navigation

Contact Info

Product

Resources

About