Asymmetric Encoder-Decoder Structured FCN Based LiDAR to Color Image Generation

Abstract

In this paper, we propose a method of generating a color image from light detection and ranging (LiDAR) 3D reflection intensity. The proposed method is composed of two steps: projection of LiDAR 3D reflection intensity into 2D intensity, and color image generation from the projected intensity by using a fully convolutional network (FCN). The color image should be generated from a very sparse projected intensity image. For this reason, the FCN is designed to have an asymmetric network structure, i.e., the layer depth of the decoder in the FCN is deeper than that of the encoder. The well-known KITTI dataset for various scenarios is used for the proposed FCN training and performance evaluation. Performance of the asymmetric network structures are empirically analyzed for various depth combinations for the encoder and decoder. Through simulations, it is shown that the proposed method generates fairly good visual quality of images while maintaining almost the same color as the ground truth image. Moreover, the proposed FCN has much higher performance than conventional interpolation methods and generative adversarial network based Pix2Pix. One interesting result is that the proposed FCN produces shadow-free and daylight color images. This result is caused by the fact that the LiDAR sensor data is produced by the light reflection and is, therefore, not affected by sunlight and shadow.

Color Image Generation from Range and Reflection Data of LiDAR

“…Interestingly, it has been reported that the deep learning-based encoder-decoder structured fully convolution network (ED-FCN) can successfully generate camera-captured-like color images from heterogeneous LiDAR reflection data [ 19 , 20 , 21 ]. Note that the ED-FCN network is originally applied for semantic segmentation.…”

“…Note that the ED-FCN network is originally applied for semantic segmentation. The methods [ 19 , 20 , 21 ] consist of two steps, as shown in Figure 1 . In the first step, LiDAR 3D reflection data are projected into 2D color image coordinate.…”

“…The projected reflection image had different sparsity and characteristics, compared with the target image. To increase better generation performance and generate color image, an asymmetric ED-FCN, i.e., a decoder, with greater depth than the encoder, was proposed in [ 20 ]. The asymmetric ED-FCN outperformed symmetric ED-FCN and the generative adversarial network (GAN)-based colorization method [ 22 ].…”

“…The SC-UNET with the connection between encoder and decoder at the two lowest levels outperforms the SC-UNET with other connections and the asymmetric ED-FCN. One interesting result discussed in [ 19 , 20 , 21 ] was that shadow-free images were generated since the LiDAR reflection data were originally produced, irrespective of the illumination. Note that these methods [ 19 , 20 , 21 ] generate an image only from the reflection data of LiDAR.…”

“…Accordingly, we propose three color-image-generation methods which fuse both reflection and range data to improve the quality of generated image—early fusion, mid fusion, and last fusion-based LiDAR to color-image-generation methods. KITTI-based evaluation dataset was used for training and performance verification [ 20 , 21 ]. Peak signal-to-noise ratio (PSNR) [ 23 ] and structural similarity index measure (SSIM) [ 23 , 24 ] metrics were used for the performance evaluation.…”