Real-Time Hybrid Multi-Sensor Fusion Framework for Perception in Autonomous Vehicles

Abstract

There are many sensor fusion frameworks proposed in the literature using different sensors and fusion methods combinations and configurations. More focus has been on improving the accuracy performance; however, the implementation feasibility of these frameworks in an autonomous vehicle is less explored. Some fusion architectures can perform very well in lab conditions using powerful computational resources; however, in real-world applications, they cannot be implemented in an embedded edge computer due to their high cost and computational need. We propose a new hybrid multi-sensor fusion pipeline configuration that performs environment perception for autonomous vehicles such as road segmentation, obstacle detection, and tracking. This fusion framework uses a proposed encoder-decoder based Fully Convolutional Neural Network (FCNx) and a traditional Extended Kalman Filter (EKF) nonlinear state estimator method. It also uses a configuration of camera, LiDAR, and radar sensors that are best suited for each fusion method. The goal of this hybrid framework is to provide a cost-effective, lightweight, modular, and robust (in case of a sensor failure) fusion system solution. It uses FCNx algorithm that improve road detection accuracy compared to benchmark models while maintaining real-time efficiency that can be used in an autonomous vehicle embedded computer. Tested on over 3K road scenes, our fusion algorithm shows better performance in various environment scenarios compared to baseline benchmark networks. Moreover, the algorithm is implemented in a vehicle and tested using actual sensor data collected from a vehicle, performing real-time environment perception.

Survey of Datafusion Techniques for Laser and Vision Based Sensor Integration for Autonomous Navigation

“…They rely on the speed and accuracy of the LiDAR for obstacle detection and camera-based identification techniques and advanced tracking and data association algorithms like Unscented Kalman Filter and Joint Probabilistic Data Association [81]. Jahromi et al proposed a real-time hybrid data fusion technique in 2019 [82]. Extended Kalman Filter (EKF) based nonlinear state estimation and encoder-decoder based Fully Convolutional Neural Network (FCNN) are used on a suite of camera, LiDAR, and radar sensors.…”

Cooperative Highway Lane Merge of Connected Vehicles Using Nonlinear Model Predictive Optimal Controller

“…Vehicle automation today is based on advanced sensor technology used to sense the environment [6]. It also depends on fast, reliable vehicle-to-vehicle (V2V) communication and intelligent control schemes (i.e., involving machine learning algorithms) to implement complex maneuvers.…”

PORF-DDPG: Learning Personalized Autonomous Driving Behavior with Progressively Optimized Reward Function

“…With the development of artificial intelligence technology [ 1 , 2 , 3 , 4 , 5 ], autonomous driving has made considerable progress. End-to-end autonomous driving vehicles, which use machine learning algorithms to generate control policies directly from sensor perception information are different from traditional autonomous driving vehicles with environment perception [ 6 , 7 ], path planning [ 8 ], and vehicle control [ 9 ]. The end-to-end autonomous driving technology is similar to the human driving mode and has attracted more and more attention for decades.…”