Pedestrian Detection Using Image Fusion and Stereo Vision in Autonomous Vehicles

shakeri, Abbas; Moshiri, Behzad; Garakani, Hossein Gharaee

doi:10.1109/istel.2018.8661069

Cited by 7 publications

(6 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The detection of pedestrians using classical methods can be found in the literature. For instance, Shakeri et al [12] use lidar and/or vision systems to enhance a region of interest (RoI) by fusing an RGB image with a depth stereo vision system for better pedestrian detection. Naoki et al [13] use a 3D lidar pointcloud to create an information map of people in motion that surrounded the vehicle.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Preliminary Study of Deep Learning Sensor Fusion for Pedestrian Detection

Plascencia

García-Gómez²,

Perez

et al. 2023

Sensors

View full text Add to dashboard Cite

Most pedestrian detection methods focus on bounding boxes based on fusing RGB with lidar. These methods do not relate to how the human eye perceives objects in the real world. Furthermore, lidar and vision can have difficulty detecting pedestrians in scattered environments, and radar can be used to overcome this problem. Therefore, the motivation of this work is to explore, as a preliminary step, the feasibility of fusing lidar, radar, and RGB for pedestrian detection that potentially can be used for autonomous driving that uses a fully connected convolutional neural network architecture for multimodal sensors. The core of the network is based on SegNet, a pixel-wise semantic segmentation network. In this context, lidar and radar were incorporated by transforming them from 3D pointclouds into 2D gray images with 16-bit depths, and RGB images were incorporated with three channels. The proposed architecture uses a single SegNet for each sensor reading, and the outputs are then applied to a fully connected neural network to fuse the three modalities of sensors. Afterwards, an up-sampling network is applied to recover the fused data. Additionally, a custom dataset of 60 images was proposed for training the architecture, with an additional 10 for evaluation and 10 for testing, giving a total of 80 images. The experiment results show a training mean pixel accuracy of 99.7% and a training mean intersection over union of 99.5%. Also, the testing mean of the IoU was 94.4%, and the testing pixel accuracy was 96.2%. These metric results have successfully demonstrated the effectiveness of using semantic segmentation for pedestrian detection under the modalities of three sensors. Despite some overfitting in the model during experimentation, it performed well in detecting people in test mode. Therefore, it is worth emphasizing that the focus of this work is to show that this method is feasible to be used, as it works regardless of the size of the dataset. Also, a bigger dataset would be necessary to achieve a more appropiate training. This method gives the advantage of detecting pedestrians as the human eye does, thereby resulting in less ambiguity. Additionally, this work has also proposed an extrinsic calibration matrix method for sensor alignment between radar and lidar based on singular value decomposition.

show abstract

Section: Introductionmentioning

confidence: 99%

“…The false negative (FN) is the number of pixels in the ground truth area that the model failed to predict. These parameters can be obtained using Equation (12). In this equation, if the confusion vector is equal to 1, this means that there is an overlapping; then the TP is computed by adding all the ones.…”

mentioning

confidence: 99%

A Preliminary Study of Deep Learning Sensor Fusion for Pedestrian Detection

Plascencia

García-Gómez²,

Perez

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…Shakeri et al. collected 3D information contained in the left-view and right-view images by a binocular stereo camera, enhanced the image quality of the pedestrian area of interest by 3D information fusion, and thus improved the accuracy of pedestrian detection [ 23 ]. However, only 2D information is used in pedestrian detection, which cannot realize real and pseudo pedestrian detection.…”

Section: Introductionmentioning

confidence: 99%

“…Nevertheless, similar to Ref. [ 23 ], only 2D information is used in pedestrian detection, which cannot complete real and pseudo pedestrian detection as well. Zhao et al.…”

Section: Introductionmentioning

confidence: 99%

Real and Pseudo Pedestrian Detection Method with CA-YOLOv5s Based on Stereo Image Fusion

Song

Yang

et al. 2022

Entropy

View full text Add to dashboard Cite

With the development of convolutional neural networks, the effect of pedestrian detection has been greatly improved by deep learning models. However, the presence of pseudo pedestrians will lead to accuracy reduction in pedestrian detection. To solve the problem that the existing pedestrian detection algorithms cannot distinguish pseudo pedestrians from real pedestrians, a real and pseudo pedestrian detection method with CA-YOLOv5s based on stereo image fusion is proposed in this paper. Firstly, the two-view images of the pedestrian are captured by a binocular stereo camera. Then, a proposed CA-YOLOv5s pedestrian detection algorithm is used for the left-view and right-view images, respectively, to detect the respective pedestrian regions. Afterwards, the detected left-view and right-view pedestrian regions are matched to obtain the feature point set, and the 3D spatial coordinates of the feature point set are calculated with Zhengyou Zhang’s calibration method. Finally, the RANSAC plane-fitting algorithm is adopted to extract the 3D features of the feature point set, and the real and pseudo pedestrian detection is achieved by the trained SVM. The proposed real and pseudo pedestrian detection method with CA-YOLOv5s based on stereo image fusion effectively solves the pseudo pedestrian detection problem and efficiently improves the accuracy. Experimental results also show that for the dataset with real and pseudo pedestrians, the proposed method significantly outperforms other existing pedestrian detection algorithms in terms of accuracy and precision.

show abstract

“…Previously, RoISC has discussed such topics, as presented by Dadet et al [5], which shows a method for reducing the error rate of vacant space in the depth data by combining a stereo camera and structure sensor. Other than that, numerous potential uses have sparked the development of stereo vision, ranging from vehicle safety [6,7], distance measurement [8], autonomous underwater vehicle [9,10], to the 3D head pose estimation [11]. However, as stereo vision cameras typically only shoot in one direction, their field of view (FOV) limited.…”

Section: Introductionmentioning

confidence: 99%

Omnidirectional Stereo Vision Study from Vertical and Horizontal Stereo Configuration

Pratama

Dewantara

Pramadihanto³

2022

EMITTER Int'l J. of Engin. Technol.

View full text Add to dashboard Cite

In stereo vision, an omnidirectional camera has high distortion compared to a standard camera, so the camera calibration is very decisive in its stereo matching. In this study, we will perform stereo matching for an omnidirectional camera with vertical and horizontal configuration so that the result of the image's depth has a 360-degree field of view by transforming the image using a calibration-based method. The result is that by using a vertical camera configuration, the image can be stereo matched directly, but by configuring a horizontal image, it is necessary to carry out a different stereo-matching process in each direction. Stereo matching with the semi-global matching method has better image results than block matching with more image objects detectable by the semi-global block matching method with a maximum disparity value of 32 pixels and with a window size of 21 pixels.

show abstract

Pedestrian Detection Using Image Fusion and Stereo Vision in Autonomous Vehicles

Cited by 7 publications

References 14 publications

A Preliminary Study of Deep Learning Sensor Fusion for Pedestrian Detection

A Preliminary Study of Deep Learning Sensor Fusion for Pedestrian Detection

Real and Pseudo Pedestrian Detection Method with CA-YOLOv5s Based on Stereo Image Fusion

Omnidirectional Stereo Vision Study from Vertical and Horizontal Stereo Configuration

Contact Info

Product

Resources

About