FlowStep3D: Model Unrolling for Self-Supervised Scene Flow Estimation

Kittenplon, Yair; Eldar, Yonina C.; Raviv, Dan

doi:10.1109/cvpr46437.2021.00410

Cited by 74 publications

(121 citation statements)

References 30 publications

Supporting

Mentioning

116

Contrasting

Order By: Relevance

“…Self-Supervised Scene Flow From Point Clouds. Some works [21,31,52] have explored unsupervised or selfsupervised learning for estimating scene flow from point clouds. To train the scene flow network without groundtruth annotation, Mittal et al [31] approximated pseudo scene flow labels based on the coordinate differences.…”

Section: Related Workmentioning

confidence: 99%

“…Alternatively, some self-supervised methods [21,24,31,52] train the scene flow network by constructing pseudo scene flow labels from point clouds. For example, Mittal et al [31] approximated pseudo scene flow labels based on the coordinate differences of 3D points, where the closest points to the next point cloud are treated as pseudo correspondence.…”

Section: Introductionmentioning

confidence: 99%

“…The training dataset only needs to additionally provide RGB images captured by an affordable monocular camera, which is more accessible compared with expensive scene flow annotation. Different from these methods [21,24,31,52], our method leverages multimodality data for generating pseudo scene flow labels.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Learning Scene Flow in 3D Point Clouds with Noisy Pseudo Labels

Li¹,

Cheng²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose a novel scene flow method that captures 3D motions from point clouds without relying on ground-truth scene flow annotations. Due to the irregularity and sparsity of point clouds, it is expensive and time-consuming to acquire ground-truth scene flow annotations. Some state-of-the-art approaches train scene flow networks in a self-supervised learning manner via approximating pseudo scene flow labels from point clouds. However, these methods fail to achieve the performance level of fully supervised methods, due to the limitations of point cloud such as sparsity and lacking color information. To provide an alternative, we propose a novel approach that utilizes monocular RGB images and point clouds to generate pseudo scene flow labels for training scene flow networks. Our pseudo label generation module infers pseudo scene labels for point clouds by jointly leveraging rich appearance information in monocular images and geometric information of point clouds. To further reduce the negative effect of noisy pseudo labels on the training, we propose a noisy-labelaware training scheme by exploiting the geometric relations of points. Experiment results show that our method not only outperforms state-of-the-art self-supervised approaches but also outperforms some supervised approaches that use accurate ground-truth flows.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning Scene Flow in 3D Point Clouds with Noisy Pseudo Labels

Li¹,

Cheng²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, Gojcic et al [14] explore weakly supervised learning for scene flow estimation using labels of ego motions as well as ground-truth foreground and background masks. Other works [62,26,35] study unsupervised/self-supervised learning for scene flow estimation on point clouds, proposing regularization losses that enforces local spatial smoothness of predicted flows. These losses are directly constraining points in a local region to have similar flows, but are not feature-aware.…”

Section: Related Workmentioning

confidence: 99%

SCTN: Sparse Convolution-Transformer Network for Scene Flow Estimation

Li¹,

Cheng²,

Giancola³

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose a novel scene flow estimation approach to capture and infer 3D motions from point clouds. Estimating 3D motions for point clouds is challenging, since a point cloud is unordered and its density is significantly non-uniform. Such unstructured data poses difficulties in matching corresponding points between point clouds, leading to inaccurate flow estimation. We propose a novel architecture named Sparse Convolution-Transformer Network (SCTN) that equips the sparse convolution with the transformer. Specifically, by leveraging the sparse convolution, SCTN transfers irregular point cloud into locally consistent flow features for estimating continuous and consistent motions within an object/local object part. We further propose to explicitly learn point relations using a point transformer module, different from exiting methods. We show that the learned relation-based contextual information is rich and helpful for matching corresponding points, benefiting scene flow estimation. In addition, a novel loss function is proposed to adaptively encourage flow consistency according to feature similarity. Extensive experiments demonstrate that our proposed approach achieves a new state of the art in scene flow estimation. Our approach achieves an error of 0.038 and 0.037 (EPE3D) on FlyingThings3D and KITTI Scene Flow respectively, which significantly outperforms previous methods by large margins.

show abstract

“…Besides, supervised methods lack generalizability while eventually only fitting domain-specific data. Self-supervised methods [25,38,62,69], on the other hand, replaced the loss between the prediction and the ground truth flow with a point distance loss to use the point cloud itself as supervision.…”

Section: Related Workmentioning

confidence: 99%

Neural Scene Flow Prior

Li¹,

Pontes²,

Lucey³

2021

Preprint

View full text Add to dashboard Cite

Before the deep learning revolution, many perception algorithms were based on runtime optimization in conjunction with a strong prior/regularization penalty. A prime example of this in computer vision is optical and scene flow. Supervised learning has largely displaced the need for explicit regularization. Instead, they rely on large amounts of labeled data to capture prior statistics, which are not always readily available for many problems. Although optimization is employed to learn the neural network, the weights of this network are frozen at runtime. As a result, these learning solutions are domain-specific and do not generalize well to other statistically different scenarios. This paper revisits the scene flow problem that relies predominantly on runtime optimization and strong regularization. A central innovation here is the inclusion of a neural scene flow prior, which uses the architecture of neural networks as a new type of implicit regularizer. Unlike learning-based scene flow methods, optimization occurs at runtime, and our approach needs no offline datasets-making it ideal for deployment in new environments such as autonomous driving. We show that an architecture based exclusively on multilayer perceptrons (MLPs) can be used as a scene flow prior. Our method attains competitive-if not better-results on scene flow benchmarks. Also, our neural prior's implicit and continuous scene flow representation allows us to estimate dense long-term correspondences across a sequence of point clouds. The dense motion information is represented by scene flow fields where points can be propagated through time by integrating motion vectors. We demonstrate such a capability by accumulating a sequence of lidar point clouds.

show abstract

FlowStep3D: Model Unrolling for Self-Supervised Scene Flow Estimation

Cited by 74 publications

References 30 publications

Learning Scene Flow in 3D Point Clouds with Noisy Pseudo Labels

Learning Scene Flow in 3D Point Clouds with Noisy Pseudo Labels

SCTN: Sparse Convolution-Transformer Network for Scene Flow Estimation

Neural Scene Flow Prior

Contact Info

Product

Resources

About