Interaction via Bi-directional Graph of Semantic Region Affinity for Scene Parsing

Ding, Henghui; Zhou, Hui; Liu, Jun; Li, Jiaxin; Feng, Zijian; Jiang, Xudong

doi:10.1109/iccv48922.2021.01555

Cited by 14 publications

(18 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2, a monocular RGB image with its depth map is taken as the input of our overall pipeline, where the depth map is optional. The object masks and their Convolutional Neural Network (CNN) features are obtained in a semantic segmentation branch [73,74]. A coarse rotation is learned by spherical learning.…”

Section: Methodsmentioning

confidence: 99%

Spatial Feature Mapping for 6DoF Object Pose Estimation

Mei¹,

Jiang²,

Ding³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

This work aims to estimate 6Dof (6D) object pose in background clutter. Considering the strong occlusion and background noise, we propose to utilize the spatial structure for better tackling this challenging task. Observing that the 3D mesh can be naturally abstracted by a graph, we build the graph using 3D points as vertices and mesh connections as edges. We construct the corresponding mapping from 2D image features to 3D points for filling the graph and fusion of the 2D and 3D features. Afterward, a Graph Convolutional Network (GCN) is applied to help the feature exchange among objects' points in 3D space. To address the problem of rotation symmetry ambiguity for objects, a spherical convolution is utilized and the spherical features are combined with the convolutional features that are mapped to the graph. Predefined 3D keypoints are voted and the 6DoF pose is obtained via the fitting optimization. Two scenarios of inference, one with the depth information and the other without it are discussed. Tested on the datasets of YCB-Video and LINEMOD, the experiments demonstrate the effectiveness of our proposed method.

show abstract

Section: Methodsmentioning

confidence: 99%

Spatial Feature Mapping for 6DoF Object Pose Estimation

Mei¹,

Jiang²,

Ding³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Hu et al [8] propose a bi-directional attention module to further utilize the features of words. Most of these works are built on FCN-like networks [29], [30] and only use the attention as auxiliary modules. Our concurrent work MDETR [31] employs DETR [15] to build an endto-end modulated detector and reason jointly over language and image.…”

Section: Referring Segmentationmentioning

confidence: 99%

VLT: Vision-Language Transformer and Query Generation for Referring Segmentation

Ding

Liu

Wang

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

We propose a Vision-Language Transformer (VLT) framework for referring segmentation to facilitate deep interactions among multi-modal information and enhance the holistic understanding to vision-language features. There are different ways to understand the dynamic emphasis of a language expression, especially when interacting with the image. However, the learned queries in existing transformer works are fixed after training, which cannot cope with the randomness and huge diversity of the language expressions. To address this issue, we propose a Query Generation Module, which dynamically produces multiple sets of input-specific queries to represent the diverse comprehensions of language expression. To find the best among these diverse comprehensions, so as to generate a better mask, we propose a Query Balance Module to selectively fuse the corresponding responses of the set of queries. Furthermore, to enhance the model's ability in dealing with diverse language expressions, we consider inter-sample learning to explicitly endow the model with knowledge of understanding different language expressions to the same object. We introduce masked contrastive learning to narrow down the features of different expressions for the same target object while distinguishing the features of different objects. The proposed approach is lightweight and achieves new state-of-the-art referring segmentation results consistently on five datasets.

show abstract

“…Semantic segmentation is a task that labels each pixel in an image by the most appropriate semantic category from the predefined ones [30,31,32,33].…”

Section: Semantic Segmentationmentioning

confidence: 99%

Self-Regularized Prototypical Network for Few-Shot Semantic Segmentation

Ding,

Zhang,

Jiang

2022

Preprint

View full text Add to dashboard Cite

The deep CNNs in image semantic segmentation typically require a large number of densely-annotated images for training and have difficulties in generalizing to unseen object categories. Therefore, few-shot segmentation has been developed to perform segmentation with just a few annotated examples. In this work, we tackle the few-shot segmentation using a self-regularized prototypical network (SRPNet) based on prototype extraction for better utilization of the support information. The proposed SRPNet extracts class-specific prototype representations from support images and generates segmentation masks for query images by a distance metric -the fidelity. A direct yet effective prototype regularization on support set is proposed in SRPNet, in which the generated prototypes are evaluated and regularized on the support set itself. The extent to which the generated prototypes restore the support mask imposes an upper limit on performance. The performance on the query set should never exceed the upper limit no matter how complete the knowledge is generalized from support set to query set. With the specific prototype regularization, SRPNet fully exploits knowledge from the support and offers high-quality prototypes that are representative for each semantic class and meanwhile discriminative for different classes. The query performance is further improved by an iterative query inference (IQI) module that combines a set of regularized prototypes. Our proposed

show abstract

Interaction via Bi-directional Graph of Semantic Region Affinity for Scene Parsing

Cited by 14 publications

References 66 publications

Spatial Feature Mapping for 6DoF Object Pose Estimation

Spatial Feature Mapping for 6DoF Object Pose Estimation

VLT: Vision-Language Transformer and Query Generation for Referring Segmentation

Self-Regularized Prototypical Network for Few-Shot Semantic Segmentation

Contact Info

Product

Resources

About