Just Go with the Flow: Self-Supervised Scene Flow Estimation

Mittal, Himangi; Okorn, Brian; Held, David

doi:10.48550/arxiv.1912.00497

Cited by 4 publications

(6 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the performance of the aforementioned methods is limited by the scale of datasets. Mittal et al [147] proposed two self-supervised losses to train their network on large unlabeled datasets. Their main idea is that a robust scene flow estimation method should be effective in both forward and backward predictions.…”

Section: D Scene Flow Estimationmentioning

confidence: 99%

Deep Learning for 3D Point Clouds: A Survey

Guo

Wang

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

1,447

590

View full text Add to dashboard Cite

Point cloud learning has lately attracted increasing attention due to its wide applications in many areas, such as computer vision, autonomous driving, and robotics. As a dominating technique in AI, deep learning has been successfully used to solve various 2D vision problems. However, deep learning on point clouds is still in its infancy due to the unique challenges faced by the processing of point clouds with deep neural networks. Recently, deep learning on point clouds has become even thriving, with numerous methods being proposed to address different problems in this area. To stimulate future research, this paper presents a comprehensive review of recent progress in deep learning methods for point clouds. It covers three major tasks, including 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation. It also presents comparative results on several publicly available datasets, together with insightful observations and inspiring future research directions.

show abstract

Section: D Scene Flow Estimationmentioning

confidence: 99%

Deep Learning for 3D Point Clouds: A Survey

Guo

Wang

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

1,447

590

View full text Add to dashboard Cite

show abstract

“…Mustafa and Hilton used semantic coherence between multiple frames to improve 4D scene flow estimation, cosegmentation and reconstruction [49]. Mittal et al [47] and PointPWCNet [80] proposed self-supervised losses to infer the scene flow in an end-to-end manner. Finally, FLOT [55] proposed a simple correspondence-based end-to-end scene flow network.…”

Section: Related Workmentioning

confidence: 99%

“…As a result, many methods have resorted to training on simulated data [44,78,55], yet this comes at the price of a non-negligible domain gap. Other methods have attempted to solve the problem in a completely unsupervised manner [72,80,47], however they fail to provide competitive performance. In Fig.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Weakly Supervised Learning of Rigid 3D Scene Flow

Gojčič¹,

Litany²,

Guibas³

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose a data-driven scene flow estimation algorithm exploiting the observation that many 3D scenes can be explained by a collection of agents moving as rigid bodies. At the core of our method lies a deep architecture able to reason at the object-level by considering 3D scene flow in conjunction with other 3D tasks. This object level abstraction, enables us to relax the requirement for dense scene flow supervision with simpler binary background segmentation mask and ego-motion annotations. Our mild supervision requirements make our method well suited for recently released massive data collections for autonomous driving, which do not contain dense scene flow annotations. As output, our model provides low-level cues like pointwise flow and higher-level cues such as holistic scene understanding at the level of rigid objects. We further propose a test-time optimization refining the predicted rigid scene flow. We showcase the effectiveness and generalization capacity of our method on four different autonomous driving datasets. We release our source code and pre-trained models under github.com/zgojcic/Rigid3DSceneFlow.

show abstract

“…In the structure-from-motion setting, learning a randomly initialized ray surface is similar to learning 3D scene flow [43], which is a challenging problem when no calibration is available, particularly when considering self-supervision [27,48]. To avoid this random initialization, we can instead learn a residual ray surface Qr , that is added to a fixed ray surface template Q 0 to produce Q = Q 0 + λ r Qr .…”

Section: Residual Ray Surface Templatementioning

confidence: 99%

Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion

Igor¹,

Guizilini²,

Ambruş³

et al. 2020

Preprint

View full text Add to dashboard Cite

Self-supervised learning has emerged as a powerful tool for depth and ego-motion estimation, leading to state-ofthe-art results on benchmark datasets. However, one significant limitation shared by current methods is the assumption of a known parametric camera model -usually the standard pinhole geometry -leading to failure when applied to imaging systems that deviate significantly from this assumption (e.g., catadioptric cameras or underwater imaging). In this work, we show that self-supervision can be used to learn accurate depth and ego-motion estimation without prior knowledge of the camera model. Inspired by the geometric model of Grossberg and Nayar, we introduce Neural Ray Surfaces (NRS), convolutional networks that represent pixel-wise projection rays, approximating a wide range of cameras. NRS are fully differentiable and can be learned end-to-end from unlabeled raw videos. We demonstrate the use of NRS for self-supervised learning of visual odometry and depth estimation from raw videos obtained using a wide variety of camera systems, including pinhole, fisheye, and catadioptric. †

show abstract

Just Go with the Flow: Self-Supervised Scene Flow Estimation

Cited by 4 publications

References 24 publications

Deep Learning for 3D Point Clouds: A Survey

Deep Learning for 3D Point Clouds: A Survey

Weakly Supervised Learning of Rigid 3D Scene Flow

Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion

Contact Info

Product

Resources

About