Integrating Millimeter Wave Radar with a Monocular Vision Sensor for On-Road Obstacle Detection Applications

Abstract

This paper presents a systematic scheme for fusing millimeter wave (MMW) radar and a monocular vision sensor for on-road obstacle detection. As a whole, a three-level fusion strategy based on visual attention mechanism and driver’s visual consciousness is provided for MMW radar and monocular vision fusion so as to obtain better comprehensive performance. Then an experimental method for radar-vision point alignment for easy operation with no reflection intensity of radar and special tool requirements is put forward. Furthermore, a region searching approach for potential target detection is derived in order to decrease the image processing time. An adaptive thresholding algorithm based on a new understanding of shadows in the image is adopted for obstacle detection, and edge detection is used to assist in determining the boundary of obstacles. The proposed fusion approach is verified through real experimental examples of on-road vehicle/pedestrian detection. In the end, the experimental results show that the proposed method is simple and feasible.

Data fusion of radar and stereo vision for detection and tracking of moving objects

“…The concept of attention windows is encountered quite often in literature regarding radarvision fusion [45][46][47][48]. In these methods, the radar provides a list of targets which are subsequently processed using image data to refine, validate or classify the target.…”

“…A decision module eventually determines the extent of the object using the output of the operators. Wang et al [46] follows a similar approach, but instead uses edge and shadow information for determining object extent. An alternative involving machine learning is presented by Ji and Prokhorov [47].…”

“…The majority of these research efforts focus on radar-guided image processing [45][46][47][48]50], feature-based fusion [49] or cross validation of detections [51,52]. Although geometric information is often estimated [49,51], kinematic and extent estimation does not follow the principles laid out in Section 2.4, but rather regards the two as decoupled entities.…”

“…Although geometric information is often estimated [49,51], kinematic and extent estimation does not follow the principles laid out in Section 2.4, but rather regards the two as decoupled entities. In addition, multi-target tracking is generally given little attention [45][46][47].…”

PELICAN: Panoramic millimeter-wave radar for perception in mobile robotics applications, Part 1: Principles of FMCW radar and of 2D image construction

“…The imaging capabilities of radars are also used for visualization and characterization of the environment [9][10][11][12], 2D/3D mapping and simultaneous localization and mapping (SLAM) applications [13][14][15]. Several researches are also involved in radar-vision fusion for obstacle detection and outdoor reconstruction [16][17][18][19] relying on the complementarity of the sensors: robustness to environmental conditions and depth detection ability of the radar; high spatial resolution of the vision. One problem for the development of radarbased solutions in mobile robotics is the radar itself, because the number of radars available on the market is not so important.…”