GAN-based Thermal Infrared Image Colorization for Enhancing Object Identification

Le-Tien, Thuong; Quang, Tran Huynh Duy; Vy, Huynh Y; Nguyen-Thanh, Tuan; Phan-Xuan, Hanh

doi:10.1109/isee51682.2021.9418801

Cited by 9 publications

(6 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, Berg et al [15] leveraged separate luminance and chrominance loss to optimize the mapping of TIR images to colored visible images. In order to increase the naturalness of the results, researchers have made more attempts [16], [17], [18] to colorize TIR images based on pixel-level content loss by introducing additional adversarial loss. However, the difficulty of collecting pixellevel aligned paired samples limits the practicality of the supervised methods for TIR colorization tasks.…”

Section: Tir Image Colorizationmentioning

confidence: 99%

Memory-Guided Collaborative Attention for Nighttime Thermal Infrared Image Colorization

Luo¹,

Cao²,

Kaifu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Nighttime thermal infrared (NTIR) image colorization, also known as translation of NTIR images into daytime color images (NTIR2DC), is a promising research direction to facilitate nighttime scene perception for humans and intelligent systems under unfavorable conditions (e.g., complete darkness). However, previously developed methods have poor colorization performance for small sample classes. Moreover, reducing the high confidence noise in pseudo-labels and addressing the problem of image gradient disappearance during translation are still under-explored, and keeping edges from being distorted during translation is also challenging. To address the aforementioned issues, we propose a novel learning framework called Memory-guided cOllaboRative atteNtion Generative Adversarial Network (MornGAN), which is inspired by the analogical reasoning mechanisms of humans. Specifically, a memory-guided sample selection strategy and adaptive collaborative attention loss are devised to enhance the semantic preservation of small sample categories. In addition, we propose an online semantic distillation module to mine and refine the pseudo-labels of NTIR images. Further, conditional gradient repair loss is introduced for reducing edge distortion during translation. Extensive experiments on the NTIR2DC task show that the proposed MornGAN significantly outperforms other image-to-image translation methods in terms of semantic preservation and edge consistency, which helps improve the object detection accuracy remarkably.

show abstract

Section: Tir Image Colorizationmentioning

confidence: 99%

Memory-Guided Collaborative Attention for Nighttime Thermal Infrared Image Colorization

Luo¹,

Cao²,

Kaifu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, Berg et al [16] realized TIR image colorization by introducing separate luminance and chrominance loss. In order to increase the naturalness of the results, more efforts [17]- [19] have focused on combining pixellevel content loss with adversarial loss to optimize the mapping from TIR to RGB images. SCGAN [20] jointly predicted the colorization and saliency map to reduce semantic confusion and color bleeding.…”

Section: A Tir Image Colorizationmentioning

confidence: 99%

“…The sampling strategy for domain B is also similar to Eq. (19). By dynamically adjusting the learning strategy through the dual feedback of both domains, the model can not only allocate learning resources more rationally (e.g., reduce redundant learning of simple samples), but also learn the features of the SOC efficiently.…”

Section: F Dual Feedback Learning Strategymentioning

confidence: 99%

Nighttime Thermal Infrared Image Colorization with Dynamic Label Mining

Luo

Cao

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Stable imaging in adverse environments (e.g., total darkness) makes thermal infrared (TIR) cameras a prevalent option for night scene perception. However, the low contrast and lack of chromaticity of TIR images are detrimental to human interpretation and subsequent deployment of RGB-based vision algorithms. Therefore, it makes sense to colorize the nighttime TIR images by translating them into the corresponding daytime color images (NTIR2DC). Despite the impressive progress made in the NTIR2DC task, how to improve the translation performance of small object classes is under-explored.To address this problem, we propose a generative adversarial network incorporating feedback-based object appearance learning (FoalGAN). Specifically, an occlusion-aware mixup module and corresponding appearance consistency loss are proposed to reduce the context dependence of object translation. As a representative example of small objects in nighttime street scenes, we illustrate how to enhance the realism of traffic light by designing a traffic light appearance loss. To further improve the appearance learning of small objects, we devise a dual feedback learning strategy to selectively adjust the learning frequency of different samples. In addition, we provide pixel-level annotation for a subset of the Brno dataset, which can facilitate the research of NTIR image understanding under multiple weather conditions. Extensive experiments illustrate that the proposed FoalGAN is not only effective for appearance learning of small objects, but also outperforms other image translation methods in terms of semantic preservation and edge consistency for the NTIR2DC task.

show abstract

“…However, under unfavorable environmental conditions, such as smoke, fog, clouds, dust, and dim light, the image quality is poor, which results in poor target feature representation ability [9]. Due to the principle of radiation imaging, infrared images can reflect the temperature distribution on the surface of an object and can obtain continuous scene image information [10]. In this case, the temperature of the object differs slightly from the surrounding environment, which results in obvious contour shape characteristics.…”

Section: Introductionmentioning

confidence: 99%

DMFFNet: Dual-Mode Multi-Scale Feature Fusion-Based Pedestrian Detection Method

Rui

Ouyang

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Most contemporary pedestrian detection algorithms are based on visible light image detection. However, in environments with dim light, small targets, and easily occluded and cluttered backgrounds, single-mode visible light images relying on color, texture, and other features cannot adequately represent the feature information of targets; as a result, a large number of targets are lost and the algorithm performance is not good. To address this problem, we propose a dual-modal multi-scale feature fusion network (DMFFNet). First, we use the MobileNet v3 backbone network to extract the features of dual-modal images as input for the multi-scale fusion attention (MFA) module, combining the idea of multi-scale feature fusion and attention mechanism. Second, we deeply fuse the multi-scale features output by the MFA with the double deep feature fusion (DDFF) module to enhance the semantic and geometric information of the target. Finally, we optimize the loss function to reflect the distance between the predicted box and the real box more realistically as well as to enhance the ability of the network toward predicting difficult samples. We performed multi-directional evaluations on the KAIST dual-light pedestrian dataset and the visible-thermal infrared pedestrian dataset (VTI) in our laboratory through comparative and ablation experiments. The overall MR -2 on the KAIST duallight pedestrian dataset is 9.26%, and the MR -2 in dim light, partial occlusion, and severe occlusion are 5.17%, 23.35%, and 47.31%, respectively. The overall MR -2 on the VIT dual-light pedestrian dataset is 9.26%, and the MR -2 in dim light, partial occlusion, and severe occlusion are 5.17%, 23.35%, and 47.31%, respectively. The results show that the algorithm performs well on pedestrian detection, especially in dim light and when the target was occluded.

show abstract

GAN-based Thermal Infrared Image Colorization for Enhancing Object Identification

Cited by 9 publications

References 16 publications

Memory-Guided Collaborative Attention for Nighttime Thermal Infrared Image Colorization

Memory-Guided Collaborative Attention for Nighttime Thermal Infrared Image Colorization

Nighttime Thermal Infrared Image Colorization with Dynamic Label Mining

DMFFNet: Dual-Mode Multi-Scale Feature Fusion-Based Pedestrian Detection Method

Contact Info

Product

Resources

About