FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentation

Deng, Fuqin; Feng, Huanqing; Liang, Mingjian; Wang, Hongmin; Yang, Yong; Gao, Yuan; Chen, Junfeng; Hu, Junjie; Guo, Xiyue; Lam, Tin Lun

doi:10.1109/iros51168.2021.9636084

Cited by 66 publications

(33 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, our proposed CMX solution, enabling comprehensive interactions from various perspectives, generalizes smoothly in RGB-T semantic segmentation. It can be seen that our method based on SegFormer-B2 achieves mIoU of 58.2%, clearly outperforming the previous best RGB-T methods ABM-DRNet [10], FEANet [17], and GMNet [19]. Our CMX solution based on SegFormer-B4 further elevates state-of-the-art mIoU to 59.7%, widening the accuracy gap in contrast to existing methods.…”

Section: Results On Rgb-thermal Datasetmentioning

confidence: 80%

“…We embed both modalities' features RGB in ∈ R H×W×C and X in ∈ R H×W×C along the spatial axis into two attention vectors W C RGB ∈ R C and W C X ∈ R C . Different from previous channel-wise attention methods [8], [17], [98], which use global average pooling to obtain the attention vector, we apply both global max pooling and global average pooling to RGB in and X in along the channel dimension to remain more information. We concatenate the four resulted vectors, having Y ∈ R 4C .…”

Section: Cross-modal Feature Rectificationmentioning

confidence: 99%

“…This is a typical strategy in RGB-D semantic segmentation with depth-guided operators [14], [15], [16], which are usually elaborately tailored for specific modalities (e.g., depth), yet hard to be extended to operate with other modality combinations. The second type of approaches [7], [8], [10], [17] deploys two backbones to perform feature extraction from RGB-and another modality separately, then fuses the extracted two features into one feature for semantic prediction (Fig. 2(b)).…”

Section: Introductionmentioning

confidence: 99%

“…This structure offers the extensibility to replace the other modalstream with alternative complementary modalities. However, most previous methods are predominantly crafted for specific modalities, e.g., they only focus on tackling RGB-D scene parsing [7], [8], [18] or RGB-T segmentation [10], [17], [19]. While inspiring, they fail to work well on different multi-modal scenarios.…”

Section: Introductionmentioning

confidence: 99%

“…Transformers handle inputs as sequences and have the ability to acquire long-range correlations [22], [25], offering the possibility for a universal framework for diverse multi-modal tasks. However, existing multi-modal fusion modules [7], [11], [17] are mostly crafted for Convolutional Neural Networks (CNNs) that fail in establishing long-range contextual dependencies in pixelrich image data, which prove essential for accurate semantic segmentation [26], [27]. Moreover, multi-modal data often encompass a great deal of noisy measurements in different sensing modalities, e.g., low-quality distance estimation regions caused by limited effective depth ranges [8] and uncertainties resulted from various event representations [12].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Liu¹,

Zhang²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

The performance of semantic segmentation of RGB images can be advanced by exploiting informative features from supplementary modalities. In this work, we propose CMX, a vision-transformer-based cross-modal fusion framework for RGB-X semantic segmentation. To generalize to different sensing modalities encompassing various uncertainties, we consider that comprehensive crossmodal interactions should be provided. CMX is built with two streams to extract features from RGB images and the complementary modality (X-modality). In each feature extraction stage, we design a Cross-Modal Feature Rectification Module (CM-FRM) to calibrate the feature of the current modality by combining the feature from the other modality, in spatial-and channel-wise dimensions. With rectified feature pairs, we deploy a Feature Fusion Module (FFM) to mix them for the final semantic prediction. FFM is constructed with a cross-attention mechanism, which enables exchange of long-range contexts, enhancing both modalities' features at a global level. Extensive experiments show that CMX generalizes to diverse multi-modal combinations, achieving state-of-the-art performances on four RGB-Depth benchmarks, as well as RGB-Thermal and RGB-Polarization datasets. Besides, to investigate the generalizability to dense-sparse data fusion, we establish a RGB-Event semantic segmentation benchmark based on the EventScape dataset, on which CMX sets the new state-of-the-art. Code is available at https://github.com/huaaaliu/RGBX Semantic Segmentation

show abstract

Section: Results On Rgb-thermal Datasetmentioning

confidence: 80%