RAFNet: RGB-D attention feature fusion network for indoor semantic segmentation

Yan, Xun-Wang; Hou, Sujuan; Karim, Awudu; Jia, Weikuan

doi:10.1016/j.displa.2021.102082

Cited by 18 publications

(6 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To validate the effectiveness of the proposed model in this paper, we compare the proposed method with state-of-the-arts methods (ESANet [24], IEMNet [60], SGACNet [61], Z-ACN [62], DynMM [63], RDFNet [7], RAFNet [64], SA-Gate [65], RedNet [8], ACNet [23], SGNet [27], ShapeConv [66]) on the NYU-Depth V2 dataset. For a fair comparison, we compare our method with others using the ResNet architecture, which employ ResNet with varying depths and quantities.…”

Section: Quantitative Experimental Results On Nyu-depth V2 and Sun Rg...mentioning

confidence: 99%

MIPANet: optimizing RGB-D semantic segmentation through multi-modal interaction and pooling attention

Zhang,

Xie

2024

Front. Phys.

View full text Add to dashboard Cite

The semantic segmentation of RGB-D images involves understanding objects appearances and spatial relationships within a scene, which necessitates careful consideration of multiple factors. In indoor scenes, the presence of diverse and disorderly objects, coupled with illumination variations and the influence of adjacent objects, can easily result in misclassifications of pixels, consequently affecting the outcome of semantic segmentation. We propose a Multi-modal Interaction and Pooling Attention Network (MIPANet) in response to these challenges. This network is designed to exploit the interactive synergy between RGB and depth modalities, aiming to enhance the utilization of complementary information and improve segmentation accuracy. Specifically, we incorporate a Multi-modal Interaction Module (MIM) into the deepest layers of the network. This module is engineered to facilitate the fusion of RGB and depth information, allowing for mutual enhancement and correction. Moreover, we introduce a Pooling Attention Module (PAM) at various stages of the encoder to enhance the features extracted by the network. The outputs of the PAMs at different stages are selectively integrated into the decoder through a refinement module to improve semantic segmentation performance. Experimental results demonstrate that MIPANet outperforms existing methods on two indoor scene datasets, NYU-Depth V2 and SUN-RGBD, by optimizing the insufficient information interaction between different modalities in RGB-D semantic segmentation. The source codes are available at https://github.com/2295104718/MIPANet.

show abstract

Section: Quantitative Experimental Results On Nyu-depth V2 and Sun Rg...mentioning

confidence: 99%

MIPANet: optimizing RGB-D semantic segmentation through multi-modal interaction and pooling attention

Zhang,

Xie

2024

Front. Phys.

View full text Add to dashboard Cite

show abstract

“…On the three evaluation metrics, CMANet achieves 74.2% pixel accuracy, 60.6% mean accuracy, and 47.6% mean IoU. Additionally, on the most important metric—mean IoU—CMANet achieves a 2.7% improvement compared to RefineNet-101 [ 45 ], 1.7% compared to LSD-GF [ 51 ], and 0.1% compared to RAFNet [ 43 ]. It is noticed that we only utilize ResNet-50 as our backbone, suggesting that CMANet is capable of better performance with a more powerful backbone.…”

Section: Methodsmentioning

confidence: 99%

“…Since the RGB and depth modalities must be fully utilized in RGB-D semantic segmentation, the cross-modality fusion is crucial. The fusion can be achieved via element-wise summation, concatenation, or a combination of both, and adapted by latent learning [ 4 , 13 , 14 , 18 , 19 , 41 , 42 , 43 ]. For learning common and specific parts from cross-modality features, ref.…”

Section: Related Workmentioning

confidence: 99%

“…Learning from the structure of RefineNet [ 45 ], RDF [ 13 ] fuses cross-modality features by applying a Multi-Modal feature Fusion (MMF) network, which is composed of several residual convolutional layers and element-wise summation. To avoid noisy and chaotic information affecting the effectiveness of the network, RAFNet [ 43 ] employs a three-stem branch encoder to process RGB, depth, and fusion features, respectively. Meanwhile, RAFNet utilizes the channel-attention model for refinement.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

CMANet: Cross-Modality Attention Network for Indoor-Scene Semantic Segmentation

Zhu

Zhang

Zhou

et al. 2022

Sensors

View full text Add to dashboard Cite

Indoor-scene semantic segmentation is of great significance to indoor navigation, high-precision map creation, route planning, etc. However, incorporating RGB and HHA images for indoor-scene semantic segmentation is a promising yet challenging task, due to the diversity of textures and structures and the disparity of multi-modality in physical significance. In this paper, we propose a Cross-Modality Attention Network (CMANet) that facilitates the extraction of both RGB and HHA features and enhances the cross-modality feature integration. CMANet is constructed under the encoder–decoder architecture. The encoder consists of two parallel branches that successively extract the latent modality features from RGB and HHA images, respectively. Particularly, a novel self-attention mechanism-based Cross-Modality Refine Gate (CMRG) is presented, which bridges the two branches. More importantly, the CMRG achieves cross-modality feature fusion and produces certain refined aggregated features; it serves as the most crucial part of CMANet. The decoder is a multi-stage up-sampled backbone that is composed of different residual blocks at each up-sampling stage. Furthermore, bi-directional multi-step propagation and pyramid supervision are applied to assist the leaning process. To evaluate the effectiveness and efficiency of the proposed method, extensive experiments are conducted on NYUDv2 and SUN RGB-D datasets. Experimental results demonstrate that our method outperforms the existing ones for indoor semantic-segmentation tasks.

show abstract

“…The feature fusion channel is to fuse the feature map features generated by the third layer of the pyramid structure. Perform a down‐sampling selection on each denoised feature map (Ge et al, 2021; Maurya et al, 2021; Yan et al, 2021), so that their sizes are the same. Then use the feature fusion model to sum each feature image by element to form a new feature map.…”

Section: Proposed Methodsmentioning

confidence: 99%

An image denoising method of picking robot vision based on feature pyramid network

Chen

Zhang

et al. 2022

J Food Process Engineering

View full text Add to dashboard Cite

Deep learning has been widely used in removing image noise, but the blind denoising problem in agriculture limits the accuracy of citrus recognition by picking robots, and most denoising methods do not preserve the texture details of the image well. To solve these problems, this study proposes a multi‐module image denoising method based on feature pyramid network. The algorithm is based on the pyramid network structure, adding modules such as attention mechanism, dilated convolution, and feature fusion to denoise images. The attention mechanism and dilated convolution enhance the extraction of noisy features. The feature fusion module fuses the feature maps of each layer to solve the problem of incomplete image detail texture preservation. The experimental results of this algorithm on different datasets show that the performance of our method on various datasets with different noise levels is superior, and it is solving the problem of real image denoising. It is effective to perform both on and save the image texture details, which proves the practicability of the method in this study. After using this method to remove the real noise in the image, the average score of citrus detection accuracy is 52.4% higher than that before denoising. Practical Applications Citrus picking is a labor‐intensive work. Traditionally, citrus picking depends on manpower. This harvesting method has high cost and low efficiency, which seriously hinders the development of citrus industry. In this study, the pyramid network structure is used to denoise the citrus image by adding modules such as attention mechanism, dilated convolution, and feature fusion, so as to guide the robot to recognize the citrus and improve the production efficiency. It is of great value to the recognition technology of industrial citrus picking robot.

show abstract

RAFNet: RGB-D attention feature fusion network for indoor semantic segmentation

Cited by 18 publications

References 22 publications

MIPANet: optimizing RGB-D semantic segmentation through multi-modal interaction and pooling attention

MIPANet: optimizing RGB-D semantic segmentation through multi-modal interaction and pooling attention

CMANet: Cross-Modality Attention Network for Indoor-Scene Semantic Segmentation

An image denoising method of picking robot vision based on feature pyramid network

Contact Info

Product

Resources

About