RGB-T Semantic Segmentation With Location, Activation, and Sharpening

Abstract

Semantic segmentation is important for scene understanding. To address the scenes of adverse illumination conditions of natural images, thermal infrared (TIR) images are introduced. Most existing RGB-T semantic segmentation methods follow three cross-modal fusion paradigms, i.e., encoder fusion, decoder fusion, and feature fusion. Some methods, unfortunately, ignore the properties of RGB and TIR features or the properties of features at different levels. In this paper, we propose a novel feature fusion-based network for RGB-T semantic segmentation, named LASNet, which follows three steps of location, activation, and sharpening. The highlight of LASNet is that we fully consider the characteristics of cross-modal features at different levels, and accordingly propose three specific modules for better segmentation. Concretely, we propose a Collaborative Location Module (CLM) for high-level semantic features, aiming to locate all potential objects. We propose a Complementary Activation Module for middle-level features, aiming to activate exact regions of different objects. We propose an Edge Sharpening Module (ESM) for lowlevel texture features, aiming to sharpen the edges of objects. Furthermore, in the training phase, we attach a location supervision and an edge supervision after CLM and ESM, respectively, and impose two semantic supervisions in the decoder part to facilitate network convergence. Experimental results on two public datasets demonstrate that the superiority of our LASNet over relevant state-of-the-art methods. The code and results of our method are available at https://github.com/MathLee/LASNet.

Semantic Segmentation Algorithm Fusing Infrared and Natural Light Images for Automatic Navigation in Transmission Line Inspection

“…Data augmentation is applied to the dataset used during training, including random flipping and cropping operations. To verify the semantic segmentation performance of the proposed method, this section compares it with six other semantic segmentation methods: BiSeNet [31], RTFNet [23], FuseSeg [24], GMNet [32], ABMDRNet [33], and LASNet [34]. Among them, BiSeNet is a semantic segmentation algorithm based on a single natural light image, while the others are based on the fusion of infrared and natural light images.…”

“…GMNet [32] only uses a deep feature fusion module to segment semantic regions, so its IOU metric is not good enough. ABMDRNet [33] and LASNet [34] employ different strategies to handle the fusion information of low-level and high-level data, so they obtained good results. In our method, distinct information is extracted for different layers, and the extracted high-level and global information is densely integrated into the restoration process of the original resolution of low-level information for each layer, so our method obtains the best result.…”

“…BiSeNet [31] only uses natural light images to segment, so it achieved a bad res RTFNet [23] and FuseSeg [24] only use element-wise summation and down-sampling/ sampling to fuse different modal images, they treat low-level and high-level informat in the same way, thus achieving relatively moderate results. GMNet [32] only uses a d feature fusion module to segment semantic regions, so its IOU metric is not good enou ABMDRNet [33] and LASNet [34] employ different strategies to handle the fusion in mation of low-level and high-level data, so they obtained good results. In our meth distinct information is extracted for different layers, and the extracted high-level a global information is densely integrated into the restoration process of the original re lution of low-level information for each layer, so our method obtains the best result.…”

Cross-Domain and Cross-Modal Knowledge Distillation in Domain Adaptation for 3D Semantic Segmentation

“…Methods Modalities % mIoU GMNet [51] RGB-Infrared 49.2 LASNet [52] RGB-Infrared 42.5 EGFNet [41] RGB-Infrared 47.3 FEANet [42] RGB-Infrared 46.8 DIDFuse [43] RGB-Infrared 50.6 ReCoNet [44] RGB-Infrared 50.9 U2Fusion [53] RGB-Infrared 47.9 TarDAL [54] RGB-Infrared 48.1 SegMiF [9] RGB-Infrared 54.8 U3M (Ours) RGB-Infrared 60.8 models utilizing RGB-Infrared modalities, a combination critical for enhancing material differentiation under varying illumination conditions. Notably, our model, U3M, achieves an impressive mIoU score of 60.8, which surpasses all other models listed.…”

Low-Light Image Enhancement by Deep Learning Network for Improved Illumination Map

“…I MAGES shot under low-light or backlit conditions are visually-terrible for viewers and also degenerate the performance of down-stream vision tasks, such as action recognition [1], [2], object detection [3], [4], and semantic segmentation [5], [6]. Many efforts have been tried to increase the visibility of these images to ameliorate their low quality, including upgrading imaging devices and designing image enhancement algorithms.…”