Augmentation of Severe Weather Impact to Far-Infrared Sensor Images to Improve Pedestrian Detection System

Abstract

Pedestrian detection is an essential task for computer vision and the automotive industry. Complex systems like advanced driver-assistance systems are based on far-infrared data sensors, used to detect pedestrians at nighttime, fog, rain, and direct sun situations. The robust pedestrian detector should work in severe weather conditions. However, only a few datasets include some examples of far-infrared images with distortions caused by atmospheric precipitation and dirt covering sensor optics. This paper proposes the deep learning-based data augmentation technique to enrich far-infrared images collected in good weather conditions by distortions, similar to those caused by bad weather. The six most accurate and fast detectors (TinyV3, TinyL3, You Only Look Once (YOLO)v3, YOLOv4, ResNet50, and ResNext50), performing faster than 15 FPS, were trained on 207,001 annotations and tested on 156,345 annotations, not used for training. The proposed data augmentation technique showed up to a 9.38 mean Average Precision (mAP) increase of pedestrian detection with a maximum of 87.02 mAP (YOLOv4). Proposed in this paper detectors’ Head modifications based on a confidence heat-map gave an additional boost of precision for all six detectors. The most accurate current detector, based on YOLOv4, reached up to 87.20 mAP during our experimental tests.

The Future of Mine Safety: A Comprehensive Review of Anti-Collision Systems Based on Computer Vision in Underground Mines

“…Moreover, the experimental findings in [104] demonstrate that CNN-based detectors achieved high performance on FIR picture at night owing to the large amount and diversity of training data. The authors of [105] propose a deep-learning-based data augmentation technique that uses the six most accurate and fastest detectors (TinyV3, TinyL3, YOLOv3, YOLOv4, ResNet50, and ResNext50) to enrich far-infrared images collected in good weather conditions with distortions similar to those caused by bad weather. Multi-spectral pictures of color thermal pairs were shown to be more successful than a single color channel for pedestrian identification in [106], particularly under difficult lighting circumstances.…”

Cause-and-Effect Analysis of ADAS: A Comparison Study between Literature Review and Complaint Data

“…Second, to address the ADAS failures caused by environmental factors, some researchers proposed machine learning models and algorithms to assist the sensors to make reasonable predictions. For example, Tumas et al [60] proposed a deep learning model to fix sensor distortion under severe weather conditions. Studies also aimed at classifying the road surface condition in order to improve the performance of Electronic Stability Control (ESC), ACC, and AEB systems [37,52].…”

Improvement of intelligent methods for pedestrian detection in far-infrared radiation images

“…In the end, conclusions are drawn and formulation of dissertations tasks are created. The review, presented in this chapter is published in three scientific papers , Tumas et al 2020and Tumas et al 2021.…”

“…The research results, presented in this chapter are published in one scientific paper (Tumas et al 2021). 51…”