Fully Automated DCNN-Based Thermal Images Annotation Using Neural Network Pretrained on RGB Data

Abstract

One of the biggest challenges of training deep neural network is the need for massive data annotation. To train the neural network for object detection, millions of annotated training images are required. However, currently, there are no large-scale thermal image datasets that could be used to train the state of the art neural networks, while voluminous RGB image datasets are available. This paper presents a method that allows to create hundreds of thousands of annotated thermal images using the RGB pre-trained object detector. A dataset created in this way can be used to train object detectors with improved performance. The main gain of this work is the novel method for fully automatic thermal image labeling. The proposed system uses the RGB camera, thermal camera, 3D LiDAR, and the pre-trained neural network that detects objects in the RGB domain. Using this setup, it is possible to run the fully automated process that annotates the thermal images and creates the automatically annotated thermal training dataset. As the result, we created a dataset containing hundreds of thousands of annotated objects. This approach allows to train deep learning models with similar performance as the common human-annotation-based methods do. This paper also proposes several improvements to fine-tune the results with minimal human intervention. Finally, the evaluation of the proposed solution shows that the method gives significantly better results than training the neural network with standard small-scale hand-annotated thermal image datasets.

Brno urban dataset: Winter extension

“…All RGB images from cameras are now accompanied by YOLOv5 detections of various objects, such as vehicles, pedestrians, bicycles, animals, and others. We have also used our algorithm [3] to transfer this information into images from the thermal camera, so all IR images are machine-annotated as well, which to our best knowledge, is unique among all other datasets now available.…”

“…For automotive applications, the most important classes are persons (0), vehicles and bikes (1–7), traffic signs (9,11), and living creatures (14–23). For IR images annotation, we used the neural network trained according to the [7] . The annotated classes are pedestrians (0), bikes (1), vehicles (2), and dogs (16).…”

“…We also included precalculated YOLOv5 [21] object detections for all RGB camera data and the IR detections by our IR-data-trained YOLO neural network [3] to the BUD:WE extension. These additions were done not only for the newly recorded data but also retroactively for both previously published sessions published by [2].…”

“…New record sessions (nearly 3 hours and 90 km) contain additional LiDAR and radar measurements on top of the old sensory equipment. Besides that, we have added DCNN (Deep Convolutional Neural Network) generated object detections and annotations of RGB images to the whole dataset (including 1 All the authors are with the Faculty of Electrical Engineering, Robotics and AI Research Group, Czechia, Brno University of Technology, Purkynova 123, Brno-Kralovo Pole, Czechia, adam.ligocki@vutbr.cz, ales.jelinek@vutbr.cz, ludek.zalud@vutbr.cz the old recordings) and even transferred these detections into IR images from the RGB camera using our algorithm [3], which makes, to our best knowledge, the BUD:WE the largest dataset of machine-annotated thermal images available online. II.…”

Thermal Feature Detection of Vehicle Categories in the Urban Area

“…Pixel classification is a topic for research in computer vision. It provides different solutions such as semi-supervised learning [25] or fully convolutional architecture [29,30].…”