Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection

Liu, Jinyuan; Fan, Xin; Huang, Zhanbo; Wu, Guanyao; Liu, Risheng; Zhong, Wei; Luo, Zhongxuan

doi:10.1109/cvpr52688.2022.00571

Cited by 329 publications

(131 citation statements)

References 30 publications

Supporting

Mentioning

131

Contrasting

Order By: Relevance

“…A. Experimental Settings 1) Datasets: We utilize the color and infrared image pairs from the MSRS [19], RoadSence [2], and M3FD datasets [80] to evaluate the proposed framework. We also compare our method with six state-of-the-art algorithms: FusionGAN [9], SDDGAN [29], GANMcC [20], SDNet [27], U2Fusion [2], and TarDAL [80].…”

Section: Methodsmentioning

confidence: 99%

“…Experimental Settings 1) Datasets: We utilize the color and infrared image pairs from the MSRS [19], RoadSence [2], and M3FD datasets [80] to evaluate the proposed framework. We also compare our method with six state-of-the-art algorithms: FusionGAN [9], SDDGAN [29], GANMcC [20], SDNet [27], U2Fusion [2], and TarDAL [80]. SDNet and U2Fusion are fusion approaches based on CNN architectures, while FusionGAN, SDDGAN, GANMcC and TarDAL are based on generative models and their variants.…”

Section: Methodsmentioning

confidence: 99%

“…The existing methods usually convert the visible images stored in three channels (i.e., RGB channels) from RGB space to YCbCr space, and use the Y channel for fusion [27], [2]. After the single-channel fused image is generated, it needs to be converted to a three-channel image through post-processing [28], [29]. Since not all channels are presented in the input data, it is hard to construct the multi-channel distribution and extract multi-channel complementary information, resulting in color distortion.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models

Yue¹,

Fang²,

Xia³

et al. 2023

Preprint

View full text Add to dashboard Cite

Color plays an important role in human visual perception, reflecting the spectrum of objects. However, the existing infrared and visible image fusion methods rarely explore how to handle multi-spectral/channel data directly and achieve high color fidelity. This paper addresses the above issue by proposing a novel method with diffusion models, termed as Dif-Fusion, to generate the distribution of the multi-channel input data, which increases the ability of multi-source information aggregation and the fidelity of colors. In specific, instead of converting multi-channel images into single-channel data in existing fusion methods, we create the multi-channel data distribution with a denoising network in a latent space with forward and reverse diffusion process. Then, we use the denoising network to extract the multi-channel diffusion features with both visible and infrared information. Finally, we feed the multi-channel diffusion features to the multi-channel fusion module to directly generate the three-channel fused image. To retain the texture and intensity information, we propose multi-channel gradient loss and intensity loss. Along with the current evaluation metrics for measuring texture and intensity fidelity, we introduce a new evaluation metric to quantify color fidelity. Extensive experiments indicate that our method is more effective than other state-of-the-art image fusion methods, especially in color fidelity. The source code will be publicly available at https://github.com/GeoVectorMatrix/Dif-Fusion.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models

Yue¹,

Fang²,

Xia³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Two datasets have been used to perform the experiments, namely M3FD [54] and RGB-NIR [55]. The first dataset is M3FD, which newly released by [54]. The M3FD dataset contains pairs of visible and thermal images.…”

Section: A Setting 1) Datasetsmentioning

confidence: 99%

TnTViT-G: Transformer in Transformer Network for Guidance Super Resolution

2023

View full text Add to dashboard Cite

Image Super Resolution is a potential approach that can improve the image quality of low-resolution optical sensors, leading to improved performance in various industrial applications. It is important to emphasize that most state-of-the-art super resolution algorithms often use a single channel of input data for training and inference. However, this practice ignores the fact that the cost of acquiring high-resolution images in various spectral domains can differ a lot from one another. In this paper, we attempt to exploit complementary information from a low-cost channel (visible image) to increase the image quality of an expensive channel (infrared image). We propose a dual stream Transformer-based super resolution approach that uses the visible image as a guide to super-resolve another spectral band image. To this end, we introduce Transformer in Transformer network for Guidance super resolution, named TnTViT-G, an efficient and effective method that extracts the features of input images via different streams and fuses them together at various stages. In addition, unlike other guidance super resolution approaches, TnTViT-G is not limited to a fixed upsample size and it can generate super-resolved images of any size. Extensive experiments on various datasets show that the proposed model outperforms other state-of-the-art super resolution approaches. TnTViT-G surpasses state-of-the-art methods by up to 0.19 ∼ 2.3dB, while it is memory efficient.

show abstract

“…In addition, the aforementioned datasets only employed monocular cameras for each sensor domain, whereas a stereo setup is more advantageous for research purposes. To the best of our knowledge, there are only two datasets [ 12 , 13 ] obtained from stereo setups for both RGB and thermal domains. However, none of these datasets pertain to intralogistics, nor do they disclose datasets.…”

Section: Introductionmentioning

confidence: 99%

Multispectral Benchmark Dataset and Baseline for Forklift Collision Avoidance

Kim

et al. 2022

Sensors

View full text Add to dashboard Cite

In this paper, multispectral pedestrian detection is mainly discussed, which can contribute to assigning human-aware properties to automated forklifts to prevent accidents, such as collisions, at an early stage. Since there was no multispectral pedestrian detection dataset in an intralogistics domain, we collected a dataset; the dataset employs a method that aligns image pairs with different domains, i.e. RGB and thermal, without the use of a cumbersome device such as a beam splitter, but rather by exploiting the disparity between RGB sensors and camera geometry. In addition, we propose a multispectral pedestrian detector called SSD 2.5D that can not only detect pedestrians but also estimate the distance between an automated forklift and workers. In extensive experiments, the performance of detection and centroid localization is validated with respect to evaluation metrics used in the driving car domain but with distinct categories, such as hazardous zone and warning zone, to make it more applicable to the intralogistics domain.

show abstract

Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection

Cited by 329 publications

References 30 publications

Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models

Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models

TnTViT-G: Transformer in Transformer Network for Guidance Super Resolution

Multispectral Benchmark Dataset and Baseline for Forklift Collision Avoidance

Contact Info

Product

Resources

About