Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation

Chen, Lin-Zhuo; Zheng, Lin; Wang, Ziqin; Yang, Yongliang; Cheng, Ming–Ming

doi:10.1109/tip.2021.3049332

Cited by 114 publications

(36 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Datasets and metrics. Among the existing RGB-D segmentation problems, the indoor semantic segmentation is rather challenging, as the objects are often complex and with severe occlusions [5]. Thus, in order to validate the effectiveness of the proposed method, we conducted experiments on three indoor RGB-D benchmarks: NYU-Depth-V2 (NYUDv2-13 and -40) [25], SUN-RGBD [26] and Stanford Indoor Dataset (SID) [1].…”

Section: Methodsmentioning

confidence: 99%

“…However, implementing such a idea is non-trivial due to the asymmetric modality problem between the RGB and the depth information. To tackle this, researchers have devoted efforts into two directions: designing dedicated architectures for RGB-D data [6,8,13,15,17,21,28], and presenting novel layers to enhance or replace the convolutional layers in RGB semantic segmentation [5,27,30]. Our method falls into the second category.…”

Section: Related Workmentioning

confidence: 99%

“…[30] presents a novel operator called mal-leable 2.5D convolution, to learn the receptive field along the depth-axis. [5] devises a S-Conv to infer the sampling offset of the convolution kernel guided by the 3D spatial information, enabling the convolutional layer to adjust the receptive field and geometric transformations. ShapeConv proposed a novel view of the content in each patch and a mechanism to leverage them adaptively with learnt weights.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation

Cao¹,

Leng²,

Lischinski³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation

Cao¹,

Leng²,

Lischinski³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The difference between appearance information and geometric information is the diverse part of these two modalities. The main focus to achieve multi-modal learning for RGB-D data is through using different fusion strategies [42,4,31,24], e.g. early fusion, late fusion or cross-level fusion.…”

Section: Multi-modal Learning With Rgb-d Datasetmentioning

confidence: 99%

RGB-D Saliency Detection via Cascaded Mutual Information Minimization

Zhang

Fan

Dai

et al. 2021

Preprint

View full text Add to dashboard Cite

Existing RGB-D saliency detection models do not explicitly encourage RGB and depth to achieve effective multimodal learning. In this paper, we introduce a novel multistage cascaded learning framework via mutual information minimization to explicitly model the multi-modal information between RGB image and depth data. Specifically, we first map the feature of each mode to a lower dimensional feature vector, and adopt mutual information minimization as a regularizer to reduce the redundancy between appearance features from RGB and geometric features from depth. We then perform multi-stage cascaded learning to impose the mutual information minimization constraint at every stage of the network. Extensive experiments on benchmark RGB-D saliency datasets illustrate the effectiveness of our framework. Further, to prosper the development of this field, we contribute the largest (7× larger than NJU2K) COME15K dataset, which contains 15,625 image pairs with high quality polygon-/scribble-/object-/instance-/rank-level annotations. Based on these rich labels, we additionally construct four new benchmarks with strong baselines and observe some interesting phenomena, which can motivate future model design. Source code and dataset are available at https://github.com/ JingZhang617/cascaded_rgbd_sod.

show abstract

“…For RGBD input, current Sconv [2] suggests learning the RGB offset from a semantic depth feature map. We share the same motivation as Sconv.…”

Section: Offset Generatormentioning

confidence: 99%

Modality-Guided Subnetwork for Salient Object Detection

Wu,

Allibert,

Stolz

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent RGBD-based models for saliency detection have attracted research attention. The depth clues such as boundary clues, surface normal, shape attribute, etc., contribute to the identification of salient objects with complicated scenarios. However, most RGBD networks require multi-modalities from the input side and feed them separately through a two-stream design, which inevitably results in extra costs on depth sensors and computation. To tackle these inconveniences, we present in this paper a novel fusion design named modality-guided subnetwork (MGSnet). It has the following superior designs: 1) Our model works for both RGB and RGBD data, and dynamically estimates depth if not available. Taking the inner workings of depthprediction networks into account, we propose to estimate the pseudo-geometry maps from RGB input -essentially mimicking the multi-modality input. 2) Our MGSnet for RGB SOD results in real-time inference but achieves stateof-the-art performance compared to other RGB models. 3) The flexible and lightweight design of MGS facilitates the integration into RGBD two-streaming models. The introduced fusion design enables a cross-modality interaction to enable further progress but with a minimal cost.

show abstract

Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation

Cited by 114 publications

References 56 publications

ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation

ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation

RGB-D Saliency Detection via Cascaded Mutual Information Minimization

Modality-Guided Subnetwork for Salient Object Detection

Contact Info

Product

Resources

About