CANet: Co-attention network for RGB-D semantic segmentation

Zhou, Hao; Qi, Lu; Huang, Hai; Xu, Yang; Wan, Zhaoliang; Wen, Xianglong

doi:10.1016/j.patcog.2021.108468

Cited by 82 publications

(16 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While previous works reach high performance on standard RGBbased semantic segmentation benchmarks, in challenging real-world conditions, it is desirable to involve multi-modality sensing for a reliable and comprehensive scene understanding [13]. RGB-Depth [73], [74], [75], [76], [77] and RGB-Thermal [19], [78], [79], [80], [81] semantic segmentation are broadly investigated. Polarimetric optical cues [11], [82], [83] and event-driven priors [12], [84], [85] are often intertwined for robust perception under adverse conditions.…”

Section: Multi-modal Semantic Segmentationmentioning

confidence: 99%

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Liu¹,

Zhang²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

The performance of semantic segmentation of RGB images can be advanced by exploiting informative features from supplementary modalities. In this work, we propose CMX, a vision-transformer-based cross-modal fusion framework for RGB-X semantic segmentation. To generalize to different sensing modalities encompassing various uncertainties, we consider that comprehensive crossmodal interactions should be provided. CMX is built with two streams to extract features from RGB images and the complementary modality (X-modality). In each feature extraction stage, we design a Cross-Modal Feature Rectification Module (CM-FRM) to calibrate the feature of the current modality by combining the feature from the other modality, in spatial-and channel-wise dimensions. With rectified feature pairs, we deploy a Feature Fusion Module (FFM) to mix them for the final semantic prediction. FFM is constructed with a cross-attention mechanism, which enables exchange of long-range contexts, enhancing both modalities' features at a global level. Extensive experiments show that CMX generalizes to diverse multi-modal combinations, achieving state-of-the-art performances on four RGB-Depth benchmarks, as well as RGB-Thermal and RGB-Polarization datasets. Besides, to investigate the generalizability to dense-sparse data fusion, we establish a RGB-Event semantic segmentation benchmark based on the EventScape dataset, on which CMX sets the new state-of-the-art. Code is available at https://github.com/huaaaliu/RGBX Semantic Segmentation

show abstract

Section: Multi-modal Semantic Segmentationmentioning

confidence: 99%

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Liu¹,

Zhang²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For the specific operation of feature fusion, some simple parameter-free operations such as concat, weighted sum or bilinear pooling [23] can build stable baseline performance. Additionally, several attention-based methods make feature fusion more flexible and learnable [1,11,14,22,49]. Moreover, alignment-based fusion methods focus on feature alignment, and they typically use flow field or deformable convolution [10,50] to align features of different levels in the spatial dimension [16,17,21,34,38].…”

Section: Feature Fusionmentioning

confidence: 99%

Changer: Feature Interaction is What You Need for Change Detection

Fang¹,

Li²,

Li³

2022

Preprint

View full text Add to dashboard Cite

Change detection is an important tool for long-term earth observation missions. It takes bi-temporal images as input and predicts "where" the change has occurred. Different from other dense prediction tasks, a meaningful consideration for change detection is the interaction between bi-temporal features. With this motivation, in this paper we propose a novel general change detection architecture, MetaChanger, which includes a series of alternative interaction layers in the feature extractor. To verify the effectiveness of MetaChanger, we propose two derived models, ChangerAD and ChangerEx with simple interaction strategies: Aggregation-Distribution (AD) and "exchange". AD is abstracted from some complex interaction methods, and "exchange" is a completely parameter&computation-free operation by exchanging bi-temporal features. In addition, for better alignment of bi-temporal features, we propose a flow dual-alignment fusion (FDAF) module which allows interactive alignment and feature fusion. Crucially, we observe Changer series models achieve competitive performance on different scale change detection datasets. Further, our proposed ChangerAD and ChangerEx could serve as a starting baseline for future MetaChanger design.

show abstract

“…CANet [58] tackles the RGB-D semantic segmentation task by proposing a co-attention network to construct a proper interaction between RGB and depth features. CANet mainly proposes a co-attention fusion module that utilizes the position and channel co-attention to adaptively fuse RGB and depth features in spatial and channel dimensions.…”

Section: Related Work A: Multi-scalementioning

confidence: 99%

“…RAN[15] proposes a Residual Attention Network, which is built by stacking complex Attention Modules to generate attention-aware features that are changed adaptively in each layer. The Attention Module used in[15] is complex because of using bottom-up and top-down feedforward structure and due to inserting trunk-and-mask attention mechanism based on hourglass modules[32] between the intermediate stages.CANet[58] tackles the RGB-D semantic segmentation task by proposing a co-attention network to construct a proper interaction between RGB and depth features. CANet mainly proposes a co-attention fusion module that utilizes the position and channel co-attention to adaptively fuse RGB and depth features in spatial and channel dimensions.…”

mentioning

confidence: 99%

EMCA: Efficient Multiscale Channel Attention Module

2022

View full text Add to dashboard Cite

Attention mechanisms have been explored with CNNs across the spatial and channel dimensions. However, all the existing methods devote the attention modules to capture local interactions from a uni-scale. This paper tackles the following question: can one consolidate multi-scale aggregation while learning channel attention more efficiently? To this end, we avail channel-wise attention over multiple feature scales, which empirically shows its aptitude to replace the limited local and uni-scale attention modules. EMCA is lightweight and can efficiently model the global context further; it is easily integrated into any feed-forward CNN architectures and trained in an end-to-end fashion. We validate our novel architecture through comprehensive experiments on image classification, object detection, and instance segmentation with different backbones. Our experiments show consistent gains in performances against their counterparts, where our proposed module, named EMCA, outperforms other channel attention techniques in accuracy and latency trade-off. More specifically, compared to SENet, we boost the accuracy by 0.8 %, 0.6 %, and 1 % on ImageNet benchmark for ResNet-18, 34, and 50, respectively. For detection and segmentation tasks, MS-COCO are for benchmarking, Our EMCA module boost the accuracy by 0.5 % and 0.3 %, respectively. We also conduct experiments that probe the robustness of the learned representations. Our code will be published once the paper is accepted.

show abstract

CANet: Co-attention network for RGB-D semantic segmentation

Cited by 82 publications

References 6 publications

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Changer: Feature Interaction is What You Need for Change Detection

EMCA: Efficient Multiscale Channel Attention Module

Contact Info

Product

Resources

About