Self-Supervised Learning of Non-Rigid Residual Flow and Ego-Motion

Tishchenko, Ivan; Lombardi, Sandro; Oswald, Martin R.; Pollefeys, Marc

doi:10.1109/3dv50981.2020.00025

Cited by 46 publications

(37 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, self-supervised learning of scene flow has important research values for 3D scene perception. Some recent works (Wu et al, 2020;Mittal et al, 2020;Pontes et al, 2020;Tishchenko et al, 2020) have been done on unsupervised learning of scene flow. PointPWC-Net (Wu et al, 2020) introduces three self-supervised losses including Chamfer loss, smoothness constraint loss, and Laplacian regularization loss in their framework for scene flow learning.…”

Section: Related Workmentioning

confidence: 99%

“…Pontes et al (Pontes et al, 2020) constrain non-rigid motion flow using graph Laplacian of raw point cloud, which embeds the topology of the point cloud to capture context information. Tishchenko et al (Tishchenko et al, 2020) divide the self-supervised learning of scene flow into two steps: ego-motion flow is calculated based on the assumption that the LiDAR is moving and the scene is stationary, and then non-rigid flow is calculated based on the assumption that the LIDAR is stationary and the scene is moving.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

SFGAN: Unsupervised Generative Adversarial Learning of 3D Scene Flow from the 3D Scene Self

Wang

Chaokang

Shen

et al. 2021

Preprint

View full text Add to dashboard Cite

3D scene flow presents the 3D motion of each point in the 3D space, which forms the fundamental 3D motion perception for autonomous driving and server robots. Although the RGBD camera or LiDAR capture discrete 3D points in space, the objects and motions usually are continuous in the macro world. That is, the objects keep themselves consistent as they flow from the current frame to the next frame. Based on this insight, the Generative Adversarial Networks (GAN) is utilized to self-learn 3D scene flow with no need for ground truth. The fake point cloud of the second frame is synthesized from the predicted scene flow and the point cloud of the first frame. The adversarial training of the generator and discriminator is realized through synthesizing indistinguishable fake point cloud and discriminating the real point cloud and the synthesized fake point cloud. The experiments on KITTI scene flow dataset show that our method realizes promising results without ground truth. Just like a human observing a real-world scene, the proposed approach is capable of determining the consistency of the scene at different moments in spite of the exact flow value of each point is unknown in advance. Corresponding author(s) Email: wanghesheng@sjtu.edu.cn

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

SFGAN: Unsupervised Generative Adversarial Learning of 3D Scene Flow from the 3D Scene Self

Wang

Chaokang

Shen

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In particular, applications such as self-driving and robot navigation rely upon a robust perception of dynamically changing 3D scenes. To equip autonomous agents with the ability to infer spatiotemporal geometric properties, there has recently been an increased interest in 3D scene flow as a form of lowlevel dynamic scene representation [37,67,73,51,49,54]. Scene flow is the 3D motion field of points in the scene [69] and is a generalization of 2D optical flow.…”

Section: Introductionmentioning

confidence: 99%

“…As a result, many methods have resorted to training on simulated data [41,73,51], yet this comes at the price of a non-negligible domain gap. Other methods have attempted to solve the problem in a completely unsupervised manner [67,75,44], however they fail to provide competitive performance. In Fig.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Weakly Supervised Learning of Rigid 3D Scene Flow

Gojčič

Litany

Guibas

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

We propose a data-driven scene flow estimation algorithm exploiting the observation that many 3D scenes can be explained by a collection of agents moving as rigid bodies. At the core of our method lies a deep architecture able to reason at the object-level by considering 3D scene flow in conjunction with other 3D tasks. This object level abstraction enables us to relax the requirement for dense scene flow supervision with simpler binary background segmentation mask and ego-motion annotations. Our mild supervision requirements make our method well suited for recently released massive data collections for autonomous driving, which do not contain dense scene flow annotations. As output, our model provides low-level cues like pointwise flow and higher-level cues such as holistic scene understanding at the level of rigid objects. We further propose a test-time optimization refining the predicted rigid scene flow. We showcase the effectiveness and generalization capacity of our method on four different autonomous driving datasets. We release our source code and pre-trained models under github.com/zgojcic/Rigid3DSceneFlow.

show abstract

SFGAN: Unsupervised Generative Adversarial Learning of 3D Scene Flow from the 3D Scene Self

Wang

Chaokang

Shen

et al. 2021

Advanced Intelligent Systems

View full text Add to dashboard Cite

Scene flow tracks the 3D motion of each point in adjacent point clouds. It provides fundamental 3D motion perception for autonomous driving and server robot. Although red green blue depth (RGBD) camera or light detection and ranging (LiDAR) capture discrete 3D points in space, the objects and motions usually are continuous in the macroworld. That is, the objects keep themselves consistent as they flow from the current frame to the next frame. Based on this insight, the generative adversarial networks (GAN) is utilized to self‐learn 3D scene flow without ground truth. The fake point cloud is synthesized from the predicted scene flow and the point cloud of the first frame. The adversarial training of the generator and discriminator is realized through synthesizing indistinguishable fake point cloud and discriminating the real point cloud and the synthesized fake point cloud. The experiments on Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset show that our method realizes promising results. Just as human, the proposed method can identify the similar local structures of two adjacent frames even without knowing the ground truth scene flow. Then, the local correspondence can be correctly estimated, and further the scene flow is correctly estimated. An interactive preprint version of the article can be found here: https://www.authorea.com/doi/full/10.22541/au.163335790.03073492.

show abstract

Self-Supervised Learning of Non-Rigid Residual Flow and Ego-Motion

Cited by 46 publications

References 19 publications

SFGAN: Unsupervised Generative Adversarial Learning of 3D Scene Flow from the 3D Scene Self

SFGAN: Unsupervised Generative Adversarial Learning of 3D Scene Flow from the 3D Scene Self

Weakly Supervised Learning of Rigid 3D Scene Flow

SFGAN: Unsupervised Generative Adversarial Learning of 3D Scene Flow from the 3D Scene Self

Contact Info

Product

Resources

About