RV-FuseNet: Range View Based Fusion of Time-Series LiDAR Data for Joint 3D Object Detection and Motion Forecasting

Abstract

Autonomous vehicles rely on robust real-time detection and future motion prediction of traffic participants to safely navigate urban environments. We present a novel end-toend approach that uses raw time-series LiDAR data to jointly solve both detection and prediction. We use the range view representation of LiDAR instead of voxelization since it does not discard information and is more efficient due to its compactness. However, for time-series fusion the data needs to be projected to a common viewpoint, and often this viewpoint is different from where it was captured leading to distortions. These distortions have an adverse impact on performance. Thus, we propose a novel architecture which reduces the impact of distortions by sequentially projecting each sweep into the viewpoint of the next sweep in time. We demonstrate that our sequential fusion approach is superior to methods that directly project all the data into the most recent viewpoint. Furthermore, we compare our approach to existing state-of-the art methods on multiple autonomous driving datasets and show competitive results.

MVFuseNet: Improving End-to-End Object Detection and Motion Forecasting through Multi-View Fusion of LiDAR Data

“…These methods, by virtue of operating in BEV, lose out on high-resolution point information and are often limited by range of operation. RV based methods such as [6] and [3] overcome the limitation on operating range but are outperformed in the motion forecasting task by recent BEV based methods. In this work, we improve the joint framework by including multi-view representation in multiple parts of the network and achieve state-of-the-art performance on both object detection and motion forecasting while scaling to large areas of operation in real-time.…”

“…For multiple views this can be trivially extended by projecting the points in both BEV and RV for aggregation. However, directly projecting all the past LiDAR data into the RV of the most recent sweep leads to significant performance degradation due to heavy data loss in the projection step [3]. Therefore, instead of previous approaches that focus on one-shot projection, we propose a novel sequential multi-view fusion approach to effectively aggregate the temporal LiDAR data.…”

“…). Similar to [3,7], if more than one point project into the same cell location l i k,n , we pick the closest point for feature rendering.…”

“…However, these methods optimize for these tasks separately, suffering from cascading errors and missing the opportunity to share learned features for both tasks [1]. To overcome these issues, multiple end-to-end methods have been proposed [2,3,4,5] for jointly solving both detection and motion forecasting. These methods have demonstrated excellent performance [1] while operating in real-time.…”

“…Most existing methods lie on the extreme ends of the spectrum with respect to when they perform this conversion during their processing, and most use a single view entirely. On one hand, methods such as [3,6,7] process LiDAR data exclusively in RV and only convert their final output to BEV during post processing. These methods are efficient for processing large spatial regions due to the compact size of the input image and offer state-of-the-art performance in the detection of small objects (e.g., pedestrians, bikes) and far away objects.…”