BANet: Bidirectional Aggregation Network With Occlusion Handling for Panoptic Segmentation

Chen, Yifeng; Lin, Guangchen; Li, Songyuan; Omar, Bourahla; Wu, Yiming; Wang, Fangfang; Feng, Junyi; Xu, Mingliang; Li, Xi

doi:10.1109/cvpr42600.2020.00385

Cited by 76 publications

(37 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Panoptic segmentation is first introduced by Kirillov et al [23], which treated countable instance things and uncountable stuff as one visual recognition task [22,23,44]. Chen et al [6] improved the panoptic segmentation quality with a bidirectional path between the semantic and instance segmentation branches. Wu et al [43] constructed modular graph structure to reason their relations.…”

Section: Related Workmentioning

confidence: 99%

GAT-CADNet: Graph Attention Network for Panoptic Symbol Spotting in CAD Drawings

Zheng¹,

Li²,

Zhu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Spotting graphical symbols from the computer-aided design (CAD) drawings is essential to many industrial applications. Different from raster images, CAD drawings are vector graphics consisting of geometric primitives such as segments, arcs, and circles. By treating each CAD drawing as a graph, we propose a novel graph attention network GAT-CADNet to solve the panoptic symbol spotting problem: vertex features derived from the GAT branch are mapped to semantic labels, while their attention scores are cascaded and mapped to instance prediction. Our key contributions are three-fold: 1) the instance symbol spotting task is formulated as a subgraph detection problem and solved by predicting the adjacency matrix; 2) a relative spatial encoding (RSE) module explicitly encodes the relative positional and geometric relation among vertices to enhance the vertex attention; 3) a cascaded edge encoding (CEE) module extracts vertex attentions from multiple stages of GAT and treats them as edge encoding to predict the adjacency matrix. The proposed GAT-CADNet is intuitive yet effective and manages to solve the panoptic symbol spotting problem in one consolidated network. Extensive experiments and ablation studies on the public benchmark show that our graph-based approach surpasses existing state-of-the-art methods by a large margin.

show abstract

Section: Related Workmentioning

confidence: 99%

GAT-CADNet: Graph Attention Network for Panoptic Symbol Spotting in CAD Drawings

Zheng¹,

Li²,

Zhu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…From the view of instance representation, previous work mainly formats things and stuff from different perspectives. Foreground things are usually separated and represented with boxes [2], [31], [32], [33] or aggregated according to center offsets [5], while background stuff is often predicted with a parallel FCN [8] branch. Although methods of [6], [23] represent things and stuff uniformly, the inherent ambiguity cannot be resolved well merely with the pixel-level affinity, which yields the performance drop in complex scenarios.…”

Section: Related Workmentioning

confidence: 99%

Fully Convolutional Networks for Panoptic Segmentation with Point-based Supervision

Li¹,

Zhao²,

Qi³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, we present a conceptually simple, strong, and efficient framework for fully-and weakly-supervised panoptic segmentation, called Panoptic FCN. Our approach aims to represent and predict foreground things and background stuff in a unified fully convolutional pipeline, which can be optimized with point-based fully or weak supervision. In particular, Panoptic FCN encodes each object instance or stuff category with the proposed kernel generator and produces the prediction by convolving the high-resolution feature directly. With this approach, instance-aware and semantically consistent properties for things and stuff can be respectively satisfied in a simple generate-kernel-then-segment workflow. Without extra boxes for localization or instance separation, the proposed approach outperforms the previous box-based and -free models with high efficiency. Furthermore, we propose a new form of pointbased annotation for weakly-supervised panoptic segmentation. It only needs several random points for both things and stuff, which dramatically reduces the annotation cost of human. The proposed Panoptic FCN is also proved to have much superior performance in this weakly-supervised setting, which achieves 82% of the fully-supervised performance with only 20 randomly annotated points per instance. Extensive experiments demonstrate the effectiveness and efficiency of Panoptic FCN on COCO, VOC 2012, Cityscapes, and Mapillary Vistas datasets. And it sets up a new leading benchmark for both fully-and weakly-supervised panoptic segmentation. Our code and models are made publicly available at https://github.com/dvlab-research/PanopticFCN.

show abstract

“…The earlier work [24] directly combines predictions of things and stuff from different models. To alleviate computation cost, many previous works [7,23,27,30,41,61,65] are proposed to model both stuff segmentation and thing segmentation in one model with different task heads. Detection based methods [20,23,31,42,62] usually represent thing within the box prediction while several bottom-up models [8,14,53,64] perform grouping instance via the pixel-level affinities or instance centers from semantic segmentation results.…”

Section: Related Workmentioning

confidence: 99%

PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic Segmentation

Yuan,

Li,

Yang

et al. 2021

Preprint

View full text Add to dashboard Cite

The recently proposed Depth-aware Video Panoptic Segmentation (DVPS) aims to predict panoptic segmentation results and depth maps in a video, which is a challenging scene understanding problem. In this paper, we present PolyphonicFormer, a vision transformer to unify all the subtasks under the DVPS task. Our method explores the relationship between depth estimation and panoptic segmentation via query-based learning. In particular, we design three different queries including thing query, stuff query, and depth query. Then we propose to learn the correlations among these queries via gated fusion. From the experiments, we prove the benefits of our design from both depth estimation and panoptic segmentation aspects. Since each thing query also encodes the instance-wise information, it is natural to perform tracking via cropping instance mask features with appearance learning. Our method ranks 1st on the ICCV-2021 BMTT Challenge video + depth track. Ablation studies are reported to show how we improve the performance. Code will be available here.

show abstract

BANet: Bidirectional Aggregation Network With Occlusion Handling for Panoptic Segmentation

Cited by 76 publications

References 31 publications

GAT-CADNet: Graph Attention Network for Panoptic Symbol Spotting in CAD Drawings

GAT-CADNet: Graph Attention Network for Panoptic Symbol Spotting in CAD Drawings

Fully Convolutional Networks for Panoptic Segmentation with Point-based Supervision

PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic Segmentation

Contact Info

Product

Resources

About