Visual Recognition by Request

Abstract

In this paper, we present a novel protocol of annotation and evaluation for visual recognition. Different from traditional settings, the protocol does not require the labeler/algorithm to annotate/recognize all targets (objects, parts, etc.) at once, but instead raises a number of recognition instructions and the algorithm recognizes targets by request. This mechanism brings two beneficial properties to reduce the burden of annotation, namely, (i) variable granularity: different scenarios can have different levels of annotation, in particular, object parts can be labeled only in large and clear instances, (ii) being open-domain: new concepts can be added to the database in minimal costs. To deal with the proposed setting, we maintain a knowledge base and design a query-based visual recognition framework that constructs queries on-the-fly based on the requests. We evaluate the recognition system on two mixed-annotated datasets, CPP and ADE20K, and demonstrate its promising ability of learning from partially labeled data as well as adapting to new concepts with only text labels.Preprint. Under review.

PanopticPartFormer++: A Unified and Decoupled View for Panoptic Part Segmentation

“…With the recently proposed backbone [92], our methods achieve the new state-of-the-art results with 63.1% PartPQ and 66.5% PWQ. Compared to the recent method using separated vision transformers [57], our methods achieve better results for both ResNet50 and a larger backbone with less GFlops and simpler pipeline. Moreover, compared to Panoptic-PartFormer, Panoptic-PartFormer++ achieve better results on all three metrics, including PQ, PartPQ and PWQ, which can be a new baseline for PPS task.…”

“…We find different backbones perform differently TABLE 3: Experiment Results on CPP. Previous works [5], [57] combine results from commonly used (top), and state-of-theart methods (bottom) for semantic segmentation, instance segmentation, panoptic segmentation, and part segmentation. Metrics split into P and NP are evaluated on scene-level classes with and without parts, respectively.…”

“…Besides, they use existing panoptic segmentation algorithms with part semantic segmentation as an isolated subnetwork. Recently, there is another work [57] formulating PPS tasks as multi-level recognition by request. However, it still uses the two separated models to handle part and things segmentation.…”

“…2, compared with previous PartPQ and HPQ, our new metric fully considers all five properties including pixel-level evaluation, regionlevel evaluation, part-whole evaluation, decouples the errors and balance the part-scene segments. Notice that recent work [57] also proposes hierarchical panoptic quality (HPQ) that can measure the accuracy of segmentation in different depths. However, it still does not well balance the ratio of part segments.…”