Robust pedestrian detection in thermal imagery using synthesized images

Abstract

In this paper we propose a method for improving pedestrian detection in the thermal domain using two stages: first, a generative data augmentation approach is used, then a domain adaptation method using generated data adapts an RGB pedestrian detector. Our model, based on the Least-Squares Generative Adversarial Network, is trained to synthesize realistic thermal versions of input RGB images which are then used to augment the limited amount of labeled thermal pedestrian images available for training. We apply our generative data augmentation strategy in order to adapt a pretrained YOLOv3 pedestrian detector to detection in the thermal-only domain. Experimental results demonstrate the effectiveness of our approach: using less than 50% of available real thermal training data, and relying on synthesized data generated by our model in the domain adaptation phase, our detector achieves state-of-the-art results on the KAIST Multispectral Pedestrian Detection Benchmark; even if more real thermal data is available adding GAN generated images to the training data results in improved performance, thus showing that these images act as an effective form of data augmentation. To the best of our knowledge, our detector achieves the best single-modality detection results on KAIST with respect to the state-of-the-art.

Partially fake it till you make it: mixing real and fake thermal images for improved object detection

“…We have also explored the effect of different quantities of augmented data w.r.t the original train set size. In Table 3 a non exhaustive analysis of the percentage of fake w.r.t real suggests, as previously noted in [38], that the optimal amount of generated data to add is between 10% and 20%, for the tested strategies increasing the amount of augmented data becomes detrimental to learning.…”

“…Similar to our approach are [30] in which a much more elaborate cycle consistent framework is developed to perform domain adaptation and [50] that improves the Cycle-GAN approach by adding per instance masks. In [38], a GAN is used to produce fake thermal images to increase the training data; the authors study the best approach on how to mix these additional data to real thermal images, showing how adding 10%-20% percent of fake images improves the performance of the object detection.…”

“…In order to achieve this we trained the generative model in a cycle-consistent setting between the source domain, the synthetic data, and the target domain, the data from FLIR dataset. The model architecture is taken from [38] as it proved to be successful in the thermal images augmentation setting and it is built using the Residual in Residual Dense Block (RRDB) as the fundamental unit. As in [44], we remove the batch normalization layer from the traditional Conv-BN-LReLU triplet.…”

“…We will use the following notation: GAN x → y to refer to an experiment in which a generative model is trained over the subset Synth x , performs inference over Synth y and finally this translated Synth y data is added to the detector training set. We also tested the generative model from [38] which performs RGB → Thermal translation, and refer to it as GAN rgb → th .…”

“…The augmentation strategy using the generative model from [38] provides some improvements compared to the baseline method but only by a 0.3% mAP.…”