Tracking Emerges by Looking Around Static Scenes, with Neural 3D Mapping

Harley, Adam W.; Lakshmikanth, Shrinidhi Kowshika; Schydlo, Paul; Fragkiadaki, Katerina

doi:10.1007/978-3-030-58574-7_36

Cited by 9 publications

(8 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, SLAM has two notable shortcomings. First, SLAM is unable to build-up an intuitive understanding of the environment (Gupta et al [ 69 ]): ‘These maps are built purely geometrically, and nothing is known until it has been explicitly observed, even when there are obvious patterns.’ New approaches therefore seek to augment SLAM with deep learning [ 70 – 74 ] (see also 3D semantic scene graphs [ 75 – 77 ]). Others seek an alternative to SLAM in deep reinforcement learning [ 69 , 78 – 83 ] or deep learning [ 84 ].…”

Section: Computer Visionmentioning

confidence: 99%

New Approaches to 3D Vision

Linton

Morgan

Read

et al. 2022

Phil. Trans. R. Soc. B

View full text Add to dashboard Cite

New approaches to 3D vision are enabling new advances in artificial intelligence and autonomous vehicles, a better understanding of how animals navigate the 3D world, and new insights into human perception in virtual and augmented reality. Whilst traditional approaches to 3D vision in computer vision (SLAM: simultaneous localization and mapping), animal navigation (cognitive maps), and human vision (optimal cue integration) start from the assumption that the aim of 3D vision is to provide an accurate 3D model of the world, the new approaches to 3D vision explored in this issue challenge this assumption. Instead, they investigate the possibility that computer vision, animal navigation, and human vision can rely on partial or distorted models or no model at all. This issue also highlights the implications for artificial intelligence, autonomous vehicles, human perception in virtual and augmented reality, and the treatment of visual disorders, all of which are explored by individual articles. This article is part of a discussion meeting issue ‘New approaches to 3D vision’.

show abstract

Section: Computer Visionmentioning

confidence: 99%

New Approaches to 3D Vision

Linton

Morgan

Read

et al. 2022

Phil. Trans. R. Soc. B

View full text Add to dashboard Cite

show abstract

“…The initial step of our model is to encode the source I s ∈ R 3×h×w with a fully convolutional U-Net encoder that produces a feature map F s ∈ R c×h×w that preserves the spatial resolution of the source image. Once a feature map F s is obtained, we perform an inverse projection step to back-project F s into a latent volumetric tensor Z s ∈ R c×ds×hs×ws , where d s , h s , w s are depth, height and width for the volumetric representation 2 . Instead of reshaping 2D feature maps into a 3D volumetric representation like ENR [13], we found that using an inverse projection step is beneficial to preserve the 3D geometry and texture information (Cf.…”

Section: Encodingmentioning

confidence: 99%

“…In order to understand the 3D world, an intelligent agent must be able to perform inference about a scene's appearance and shape from unseen viewpoints. Being able to synthesize images at target camera viewpoints efficiently given sparse source views serves a fundamental purpose in building intelligent visual behaviour [2,3,4]. The problem of learning to synthesize novel views has been widely studied in literature, with approaches ranging from traditional small-baseline view synthesis relying on multi-plane imaging [5,6,7,8], flow estimation [9,10], to explicitly modeling 3D geometry via point-clouds [11], meshes [12], and voxels [13].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Fast and Explicit Neural View Synthesis

Guo¹,

Bautista²,

Colburn³

et al. 2021

Preprint

View full text Add to dashboard Cite

We study the problem of novel view synthesis of a scene comprised of 3D objects. We propose a simple yet effective approach that is neither continuous nor implicit, challenging recent trends on view synthesis. We demonstrate that although continuous radiance field representations have gained a lot of attention due to their expressive power, our simple approach obtains comparable or even better novel view reconstruction quality comparing with state-of-the-art baselines while increasing rendering speed by over 400x. Our model is trained in a category-agnostic manner and does not require scene-specific optimization. Therefore, it is able to generalize novel view synthesis to object categories not seen during training. In addition, we show that with our simple formulation, we can use view synthesis as a self-supervision signal for efficient learning of 3D geometry without explicit 3D supervision.Preprint. Under review.

show abstract

“…Many of the environments are photo-realistic reconstructions of indoor [52,3,60,9] and outdoor [15,19] scenes, and provide 3D ground truth labels for objects. These simulated environments have been used to study tasks such as visual navigation and exploration [10,21,18], visual question answering [14], tracking [22], and object recognition [11,62]. In our work, we use a simulated embodied agent to discover objects and fixate their sensors on them to obtain object-centric data for fine-tuning a detector.…”

Section: Active Visual Learningmentioning

confidence: 99%

Move to See Better: Self-Improving Embodied Object Detection

Fang,

Jain,

Sarch

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Humans learn to better understand the world by moving around their environment to get more informative viewpoints of the scene. Most methods for 2D visual recognition tasks such as object detection and segmentation treat images of the same scene as individual samples and do not exploit object permanence in multiple views. Generalization to novel scenes and views thus requires additional training with lots of human annotations. In this paper, we propose a self-supervised framework to improve an object detector in unseen scenarios by moving an agent around in a 3D environment and aggregating multi-view RGB-D information. We unproject confident 2D object detections from the pre-trained detector and perform unsupervised 3D segmentation on the point cloud. The segmented 3D objects are then re-projected to all other views to obtain pseudo-labels for fine-tuning. Experiments on both indoor and outdoor datasets show that (1) our framework performs high quality 3D segmentation from raw RGB-D data and a pre-trained 2D detector; (2) fine-tuning with self supervision improves the 2D detector significantly where an unseen RGB image is given as input at test time; (3) training a 3D detector with self supervision outperforms a comparable self-supervised method by a large margin.

show abstract

Tracking Emerges by Looking Around Static Scenes, with Neural 3D Mapping

Cited by 9 publications

References 38 publications

New Approaches to 3D Vision

New Approaches to 3D Vision

Fast and Explicit Neural View Synthesis

Move to See Better: Self-Improving Embodied Object Detection

Contact Info

Product

Resources

About