ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints

Shah, Dhruv; Levine, Sergey

doi:10.48550/arxiv.2202.11271

Cited by 4 publications

(8 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Others proposed similar models to synthesize control images to maximally activate specific neuron sites in the monkey V4 ( 54 ). In a somewhat related robotics study, a deep learning network used the robot’s local views and geographic hints, such as satellite images or road maps, to plan paths over a variety of environments ( 55 ). In the present work, these TDVs were used to predict FPV, and vice versa.…”

Section: Discussionmentioning

confidence: 99%

Linking global top-down views to first-person views in the brain

Xing

Chrastil

Nitz

et al. 2022

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Humans and other animals have a remarkable capacity to translate their position from one spatial frame of reference to another. The ability to seamlessly move between top-down and first-person views is important for navigation, memory formation, and other cognitive tasks. Evidence suggests that the medial temporal lobe and other cortical regions contribute to this function. To understand how a neural system might carry out these computations, we used variational autoencoders (VAEs) to reconstruct the first-person view from the top-down view of a robot simulation, and vice versa. Many latent variables in the VAEs had similar responses to those seen in neuron recordings, including location-specific activity, head direction tuning, and encoding of distance to local objects. Place-specific responses were prominent when reconstructing a first-person view from a top-down view, but head direction–specific responses were prominent when reconstructing a top-down view from a first-person view. In both cases, the model could recover from perturbations without retraining, but rather through remapping. These results could advance our understanding of how brain regions support viewpoint linkages and transformations.

show abstract

Section: Discussionmentioning

confidence: 99%

Linking global top-down views to first-person views in the brain

Xing

Chrastil

Nitz

et al. 2022

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…Therefore, it is generally only effective for short-horizon goals. In the case of navigation tasks studied in prior work with such approaches, this typically means goals that are within line of sight of the robot, or within a few tens of metres of its present location [6,[38][39][40]64,75], though some works have explored extensions to enable significantly longer-range control in some settings, including through the use of memory and recurrence [48,51]. As a side note, o t in general may not represent a Markovian state of the system, but only an observation.…”

Section: Learning Policies From Datamentioning

confidence: 99%

“…In general, observations in a new environment might be added to the robot's memory ('mental map'), connecting them to a growing graph, and each time the robot might replan a path toward the goal. When the path to the goal cannot be determined because the environment has not been explored sufficiently, the robot might choose to explore a new location [39], or might use some sort of heuristic informed by side information, such as the spatial coordinate of the target, or even an overhead map [40]. The latter also provides a natural avenue for introducing other goal specification modalities: while the mental map is built in terms of the robot's observations, the final goal can be specified in terms of any function of this observation, including potentially its GPS coordinates [40].…”

Section: Planning and High-level Decision Makingmentioning

confidence: 99%

“…Methods based on human-provided demonstrations (4), which have a long history in robotic navigation [23][24][25][26][27][28]35,36], have the benefit of learning about the world as it really is, but carry a heavy price: the performance of the system is entirely limited by the number of demonstrations that are provided and does not improve with more use. In contrast, experiential learning methods [6,[37][38][39][40], which may also utilize demonstration data in combination with the robot's own experience and, crucially, do not make the assumption that all of the provided data is good (i.e. it should not be imitated blindly), offer the most appealing combination of benefits.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning robotic navigation from experience: principles, methods and recent results

Levine

Shah

2022

Phil. Trans. R. Soc. B

View full text Add to dashboard Cite

Navigation is one of the most heavily studied problems in robotics and is conventionally approached as a geometric mapping and planning problem. However, real-world navigation presents a complex set of physical challenges that defies simple geometric abstractions. Machine learning offers a promising way to go beyond geometry and conventional planning, allowing for navigational systems that make decisions based on actual prior experience. Such systems can reason about traversability in ways that go beyond geometry, accounting for the physical outcomes of their actions and exploiting patterns in real-world environments. They can also improve as more data is collected, potentially providing a powerful network effect. In this article, we present a general toolkit for experiential learning of robotic navigation skills that unifies several recent approaches, describe the underlying design principles, summarize experimental results from several of our recent papers, and discuss open problems and directions for future work. This article is part of the theme issue ‘New approaches to 3D vision’.

show abstract

“…Using supervised contrastive learning, Gao et al [14] manually label a set of anchor patches in their effort to efficiently create a feature representation that is able to distinguish different traversability regions. In [62], the output of an heuristic model trained on teleoperated prior data using the contrastive InfoNCE loss function [75], is combined with the output of a local traversability model towards successful path planning.…”

Section: Unsupervised and Semi-supervisedmentioning

confidence: 99%

A Survey of Traversability Estimation for Mobile Robots

Sevastopoulos

Konstantopoulos

2022

IEEE Access

View full text Add to dashboard Cite

Traversability illustrates the difficulty of driving through a specific region and encompasses the suitability of the terrain for traverse based on its physical properties, such as slope and roughness, surface condition, etc. In this survey we highlight the merits and limitations of all the major steps in the evolution of traversability estimation techniques, covering both non-trainable and machine-learning methods, leading up to the recent proliferation of deep learning literature. We discuss how the nascence of Deep Learning has created an opportunity for radical improvement in traversability estimation. Finally, we discuss how selfsupervised learning can help satisfy deep methods' increased need for (challenging to acquire and label) large-scale datasets.

show abstract

ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints

Cited by 4 publications

References 34 publications

Linking global top-down views to first-person views in the brain

Linking global top-down views to first-person views in the brain

Learning robotic navigation from experience: principles, methods and recent results

A Survey of Traversability Estimation for Mobile Robots

Contact Info

Product

Resources

About