BE-STI: Spatial-Temporal Integrated Network for Class-agnostic Motion Prediction with Bidirectional Enhancement

Wang, Yunlong; Pan, Hongyu; Zhu, Jun; Wu, Yu-Huan; Zhan, Xin; Jiang, Kun; Yang, Diange

doi:10.1109/cvpr52688.2022.01658

Cited by 15 publications

(22 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Motion prediction aims to estimate the future positions of objects based on past observations. Given consecutive point clouds from past frames, some works [28], [29], [30], [31], [58], [59], [60] propose to convert the point clouds into bird's eye view (BEV) maps and estimate a future motion field from these BEV maps. MotionNet [28] learns to simultaneously estimate both semantic information and future motion in a supervised manner.…”

Section: Related Workmentioning

confidence: 99%

“…We conduct motion prediction experiments on nuScenes [73]. Following previous works [28], [29], [30], [31], we divide the dataset into three parts: 500 scenes for training, 100 for validation, and 250 for test. During the validation and testing phases, the ground truth motion data is derived from the detection and tracking annotations provided by nuScenes.…”

Section: Application On Self-supervised Class-agnostic Motion Predictionmentioning

confidence: 99%

“…Implementation details. During the evaluation of our method for motion prediction, we follow [28], [30] to prepare data. Specifically, we crop each input point cloud within the spatial bounds of [−32, 32] × [−32, 32] × [−3, 2] meters and subsequently partition the input data into voxels with the shape of (0.25, 0.25, 0.4) meters.…”

Section: Application On Self-supervised Class-agnostic Motion Predictionmentioning

confidence: 99%

“…Class-agnostic motion prediction aims to generate the future positions of objects based on previous observations, which holds significance for path planning and navigation. Given a series of point clouds from past frames, some works [28], [29], [30], [31] propose to convert the past point clouds into bird's eye view (BEV) maps and output a motion vector of each cell in the current BEV map to indicate its displacement from the current to the future. Treating the BEV-based motion field as scene flow, we generate pseudo BEV-based motion labels via our self-supervised scene flow method and use these pseudo labels to train motion prediction models.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

RigidFlow: Self-Supervised Scene Flow Learning on Point Clouds by Local Rigidity Prior

Zhang

Lin

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

In this article, we investigate self-supervised 3D scene flow estimation and class-agnostic motion prediction on point clouds. A realistic scene can be well modeled as a collection of rigidly moving parts, therefore its scene flow can be represented as a combination of the rigid motion of these individual parts. Building upon this observation, we propose to generate pseudo scene flow labels for self-supervised learning through piecewise rigid motion estimation, in which the source point cloud is decomposed into local regions and each region is treated as rigid. By rigidly aligning each region with its potential counterpart in the target point cloud, we obtain a region-specific rigid transformation to generate its pseudo flow labels. To mitigate the impact of potential outliers on label generation, when solving the rigid registration for each region, we alternately perform three steps: establishing point correspondences, measuring the confidence for the correspondences, and updating the rigid transformation based on the correspondences and their confidence. As a result, confident correspondences will dominate label generation and a validity mask will be derived for the generated pseudo labels. By using the pseudo labels together with their validity mask for supervision, models can be trained in a self-supervised manner. Extensive experiments on FlyingThings3D and KITTI datasets demonstrate that our method achieves new state-of-the-art performance in self-supervised scene flow learning, without any ground truth scene flow for supervision, even performing better than some supervised counterparts. Additionally, our method is further extended to class-agnostic motion prediction and significantly outperforms previous state-of-the-art self-supervised methods on nuScenes dataset.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Application On Self-supervised Class-agnostic Motion Predictionmentioning

confidence: 99%

Section: Application On Self-supervised Class-agnostic Motion Predictionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

RigidFlow: Self-Supervised Scene Flow Learning on Point Clouds by Local Rigidity Prior

Zhang

Lin

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…However, these approaches may face challenges when handling categories that have not been seen in the training set, mainly due to their reliance on object detection (Wu, Chen, and Metaxas 2020). To address this challenge, class-agnostic motion prediction task (Schreiber, Hoermann, and Dietmayer 2019;Wu, Chen, and Metaxas 2020;Wang et al 2022;Wei et al 2022) is proposed to provide complementary information. These methods take a sequence of previous point clouds as input and predict the future displacements for each Bird's Eye View (BEV) cell.…”

Section: Introductionmentioning

confidence: 99%

Key Technology of Automation Control Based on Artificial Intelligence Technology

Wang

2021

Advances in Intelligent Systems and Computing

View full text Add to dashboard Cite

Current state-of-the-art approaches for few-shot action recognition achieve promising performance by conducting frame-level matching on learned visual features. However, they generally suffer from two limitations: i) the matching procedure between local frames tends to be inaccurate due to the lack of guidance to force long-range temporal perception; ii) explicit motion learning is usually ignored, leading to partial information loss. To address these issues, we develop a Motion-augmented Long-short Contrastive Learning (MoLo) method that contains two crucial components, including a long-short contrastive objective and a motion autodecoder. Specifically, the long-short contrastive objective is to endow local frame features with long-form temporal awareness by maximizing their agreement with the global token of videos belonging to the same class. The motion autodecoder is a lightweight architecture to reconstruct pixel motions from the differential features, which explicitly embeds the network with motion dynamics. By this means, MoLo can simultaneously learn long-range temporal context and motion cues for comprehensive few-shot matching. To demonstrate the effectiveness, we evaluate MoLo on five standard benchmarks, and the results show that MoLo favorably outperforms recent advanced methods. The source code is available at https://github. com/alibaba-mmai-research/MoLo.

show abstract

Anchor-Free Spatio-Temporal Attention Pyramid Network for Object Detection in Driving Scenes

Liu,

Si,

Liu

et al. 2023

2023 China Automation Congress (CAC)

View full text Add to dashboard Cite

BE-STI: Spatial-Temporal Integrated Network for Class-agnostic Motion Prediction with Bidirectional Enhancement

Cited by 15 publications

References 34 publications

RigidFlow: Self-Supervised Scene Flow Learning on Point Clouds by Local Rigidity Prior

RigidFlow: Self-Supervised Scene Flow Learning on Point Clouds by Local Rigidity Prior

Key Technology of Automation Control Based on Artificial Intelligence Technology

Anchor-Free Spatio-Temporal Attention Pyramid Network for Object Detection in Driving Scenes

Contact Info

Product

Resources

About