Keypoints-Based Deep Feature Fusion for Cooperative Vehicle Detection of Autonomous Driving

Abstract

Sharing collective perception messages (CPM) between vehicles is investigated to decrease occlusions so as to improve the perception accuracy and safety of autonomous driving. However, highly accurate data sharing and low communication overhead is a big challenge for collective perception, especially when real-time communication is required among connected and automated vehicles. In this letter, we propose an efficient and effective keypoints-based deep feature fusion framework built on the 3D object detector PV-RCNN, called Fusion PV-RCNN (FPV-RCNN for short), for collective perception. We introduce a high-performance bounding box proposal matching module and a keypoints selection strategy to compress the CPM size and solve the multi-vehicle data fusion problem. Besides, we also propose an effective localization error correction module based on the maximum consensus principle to increase the robustness of the data fusion. Compared to a bird's-eye view (BEV) keypoints feature fusion, FPV-RCNN achieves improved detection accuracy by about 9% at a high evaluation criterion (IoU 0.7) on the synthetic dataset COMAP dedicated to collective perception. In addition, its performance is comparable to two raw data fusion baselines that have no data loss in sharing. Moreover, our method also significantly decreases the CPM size to less than 0.3 KB, and is thus about 50 times smaller than the BEV feature map sharing used in previous works. Even with further decreased CPM feature channels, i. e., from 128 to 32, the detection performance does not show apparent drops. The code of our method is available at https://github.com/YuanYunshuang/FPV_RCNN.

Collaborative Perception in Autonomous Driving: Methods, Datasets and Challenges

“…The early and late collaboration fusion targets have clear physical meanings, and simple fusion and data compression strategies are sufficient [24,41,51]. In intermediate collaboration, considering deep semantic information contained in the shared features, the reasonable feature selection [52,53] and fusion strategies [19,20,21] are required. Besides, the collaboration among agents will bring redundant and uncertain information.…”

V2XP-ASG: Generating Adversarial Scenes for Vehicle-to-Everything Perception

“…V2X Perception: V2X perception investigates how to leverage the visual information from nearby AVs and intelligent infrastructure to enhance the perception capability. Based on the collaboration strategies, there are three major classes: early [21], late [22], [23], [24], and intermediate fusion [5], [7], [10], [4], [25], [26], [27]. The early fusion method delivers the raw point clouds across agents, and each agent will feed the aggregated point clouds to the network for 3D detection.…”

“…Sooner or later, these autonomous systems will be deployed on roads at scale, opening up opportunities for cooperation between them. Previous works in [4], [5], [6], [7], [8], [9], [10], [11], [12] have demonstrated that by leveraging the Vehicle-to-Everything (V2X) communication technology, AVs and infrastructure can perform cooperative perception by using the shared sensing information and thus significantly enhance the perception performance. Despite the remarkable improvement, these works evaluate the proposed systems on the dataset with natural scenarios that do not contain sufficient safety-critical scenes.…”

An Efficient and Robust Object-Level Cooperative Perception Framework for Connected and Automated Driving

“…However, these existing studies are vulnerable to location and pose errors which are common and inevitable in real-world applications. FPV-RCNN [2] tried to introduce a location error correction module based on keypoints matching before feature fusion to make the model more robust, but it can not handle larger errors. Vadivelu et alproposed a deep learning based framework to estimate potential errors [14], but they rely on feature-level fusion, which requires high computational capacity and is not general among different scenarios.…”

“…The primary bottleneck for cooperative perception is the sharing of precise data with low latency and low communication burden [2]. Generally, sharing raw data provides the best performance because of the slightest information loss.…”