Part-aware Panoptic Segmentation

Geus, Daan de; Meletis, Panagiotis; Lu, Chenyang; Wen, Xiaoxiao; Dubbelman, Gijs

doi:10.1109/cvpr46437.2021.00544

Cited by 34 publications

(29 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We establish our benchmark on the Cityscapes-Panoptic-Parts (CPP) dataset [13] and the ADE20K dataset [65]. The knowledge base for CPP contains 19 object and 9 non-duplicate part classes, while that for ADE20K has 150 object and 83 part classes which appear frequently.…”

Section: Knowledge Base Whole-part Hierarchymentioning

confidence: 99%

“…The system fetches the class label, c l , and looks up the knowledge base for the set of its sub-classes in texts (e.g., in the CPP dataset [13], person has sub-classes of head, torso, arm, and leg). The task is to segment the mask, M l , into sub-masks -each pixel can belong to one of the specified sub-classes or not (i.e., a reject option is available for each pixel).…”

Section: Problem Formulationmentioning

confidence: 99%

“…We evaluate visual recognition by request on two datasets. (1) The Cityscapes-Panoptic-Parts (CPP) dataset [13] inherited the images and 19 object classes from the Cityscapes dataset [11], and further annotated two groups of part classes for vehicles and persons. As a result, there are 5 object classes having parts, with 9 non-duplicate part classes in total.…”

Section: Dataset and The Evaluation Metricmentioning

confidence: 99%

See 2 more Smart Citations

Visual Recognition by Request

Tang¹,

Xie²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we present a novel protocol of annotation and evaluation for visual recognition. Different from traditional settings, the protocol does not require the labeler/algorithm to annotate/recognize all targets (objects, parts, etc.) at once, but instead raises a number of recognition instructions and the algorithm recognizes targets by request. This mechanism brings two beneficial properties to reduce the burden of annotation, namely, (i) variable granularity: different scenarios can have different levels of annotation, in particular, object parts can be labeled only in large and clear instances, (ii) being open-domain: new concepts can be added to the database in minimal costs. To deal with the proposed setting, we maintain a knowledge base and design a query-based visual recognition framework that constructs queries on-the-fly based on the requests. We evaluate the recognition system on two mixed-annotated datasets, CPP and ADE20K, and demonstrate its promising ability of learning from partially labeled data as well as adapting to new concepts with only text labels.Preprint. Under review.

show abstract

Section: Knowledge Base Whole-part Hierarchymentioning

confidence: 99%

Section: Problem Formulationmentioning

confidence: 99%

Section: Dataset and The Evaluation Metricmentioning

confidence: 99%

See 1 more Smart Citation

Visual Recognition by Request

Tang¹,

Xie²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…One way to deal with this is to learn the panoptic segmentation based on the semantic and instance logits in a parameter-free fashion as a post-processing fusion technique [4], [16], which, however, does not provide an estimate on the prediction uncertainty. Moreover, the proposed extensions to the panoptic segmentation task also do not provide any uncertainty estimation [17], [18], [19].…”

Section: A Panoptic Segmentationmentioning

confidence: 99%

Uncertainty-aware Panoptic Segmentation

Sirohi¹,

Marvi²,

Büscher³

et al. 2022

Preprint

View full text Add to dashboard Cite

Reliable scene understanding is indispensable for modern autonomous systems. Current learning-based methods typically try to maximize their performance based on segmentation metrics that only consider the quality of the segmentation. However, for the safe operation of a system in the real world it is crucial to consider the uncertainty in the prediction as well. In this work, we introduce the novel task of uncertainty-aware panoptic segmentation, which aims to predict per-pixel semantic and instance segmentations, together with per-pixel uncertainty estimates. We define two novel metrics to facilitate its quantitative analysis, the uncertaintyaware Panoptic Quality (uPQ) and the panoptic Expected Calibration Error (pECE). We further propose the novel topdown Evidential Panoptic Segmentation Network (EvPSNet) to solve this task. Our architecture employs a simple yet effective probabilistic fusion module that leverages the predicted uncertainties. Additionally, we propose a new Lovász evidential loss function to optimize the IoU for the segmentation utilizing the probabilities provided by deep evidential learning. Furthermore, we provide several strong baselines combining state-of-the-art panoptic segmentation networks with samplingfree uncertainty estimation techniques. Extensive evaluations show that our EvPSNet achieves the new state-of-the-art for the standard Panoptic Quality (PQ), as well as for our uncertaintyaware panoptic metrics.

show abstract

“…Pixel-based scene understanding Scene understanding is vital for intelligent vehicles, which has also been a broad and challenging topic in the computer vision community. Various sub-tasks with corresponding solutions are proposed step-by-step towards a more comprehensive understanding of images: from object classification [16] to detection [17], and from semantic segmentation [2] to part-aware panoptic segmentation [18]. Each evolutionary step introduced extra information that a network could deliver, and took a step forward towards a more holistic understanding of scenes.…”

Section: Related Workmentioning

confidence: 99%