Multimodal Fusion Object Detection System for Autonomous Vehicles

Abstract

In order for autonomous vehicles to safely navigate the road ways, accurate object detection must take place before safe path planning can occur. Currently, general purpose object detection convolutional neural network (CNN) models have the highest detection accuracies of any method. However, there is a gap in the proposed detection frameworks. Specifically, those that provide high detection accuracy necessary for deployment but do not perform inference in realtime, and those that perform inference in realtime but detection accuracy is low. We propose multimodel fusion detection system (MFDS), a sensor fusion system that combines the speed of a fast image detection CNN model along with the accuracy of light detection and range (LiDAR) point cloud data through a decision tree approach. The primary objective is to bridge the tradeoff between performance and accuracy. The motivation for MFDS is to reduce the computational complexity associated with using a CNN model to extract features from an image. To improve efficiency, MFDS extracts complimentary features from the LiDAR point cloud in order to obtain comparable detection accuracy. MFDS is novel by not only using the image detections to aid three-dimensional (3D) LiDAR detection but also using the LiDAR data to jointly bolster the image detections and provide 3D detections. MFDS achieves 3.7% higher accuracy than the base CNN detection model and is able to operate at 10 Hz. Additionally, the memory requirement for MFDS is small enough to fit on the Nvidia Tx1 when deployed on an embedded device.

Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines

“…Multimodal deep learning models that can ingest pixel data along with other data types (fusion) have been successful in applications outside of medicine, such as autonomous driving and video classification. As an example, a multimodal fusion detection system for autonomous vehicles, that combines visual features from cameras along with data from Light Detection and Ranging (LiDAR) sensors, is able to achieve significantly higher accuracy (3.7% improvement) than a single-modal CNN detection model 21 . Similarly, a multimodal social media video classification pipeline leveraging both visual and textual features increased the classification accuracy to 88.0%, well above single modality neural networks such as Google’s InceptionV3 which reached an accuracy of 76.4% on the same task 22 .…”

Point-cloud based 3D object detection and classification methods for self-driving applications: A survey and taxonomy

“…Representative examples of such methodologies are the Sliding Shapes [23], Vote3D [24], VoxelNet [25], MFDS [26], MEGVII [27], 3D FCN [28], Vote3Deep [24], SECOND [29], Patch Refinement [30], Fast Point R-CNN [31], Voxel-FPN [32], PV-RCNN [33], HotSpotNet [67,3DBN [34], Fusion of Fusion Net [35] and Point A 2 NET [36].…”

“…Several methods, such as Frustum PointNet (F-PointNet) [46], Multi-View 3D Object Detection (MV3D) [51], PointFusion [43], Aggregate View Object Detection network (AVOD) [56], RegiOn Approximation Refinement Network (Roarnet) [44], ContFuse [58], Multimodal Fusion Detection System (MFDS) [26], Multi-task Multi-sensor Fusion (MMF) [59], SCANet [57], PointPainting [45], SIFRNet [48], Complex-Retina [60] and LaserNet++ [61] propose a combination of images and LIDAR data to improve object detection accuracy. It allows for performing object detection in difficult scenarios, such as classifying small objects (pedestrians, cyclists) or distant objects, which is one of the limitations found in LiDAR-based object detection models.…”

“…Instead of using 2D regions to propose frustums, RoarNet [44] uses a RoarNet 2D network based on 2D CNNs, to propose 3D regions. MFDS also uses 2D object detectors to propose a set of possible objects, but they pair these detections with LiDAR point cloud clusters, which are formed based on Euclidean distance [26]. The approaches presented here, typically use PointNet++ or PointNet to learn point cloud features and segment object instances.…”

CNN based sensor fusion method for real-time autonomous robotics systems

“…The remaining points in the point cloud were divided into clusters according to the euclidean distance. Thus, the length of the object, its distance from the center and its direction were determined [26]. Babak et al proposed a new multi-sensor fusion pipeline configuration for object detection and tracking.…”