Learning What Not to Segment: A New Perspective on Few-Shot Segmentation

Abstract

Recently few-shot segmentation (FSS) has been extensively developed. Most previous works strive to achieve generalization through the meta-learning framework derived from classification tasks; however, the trained models are biased towards the seen classes instead of being ideally class-agnostic, thus hindering the recognition of new concepts. This paper proposes a fresh and straightforward insight to alleviate the problem. Specifically, we apply an additional branch (base learner) to the conventional FSS model (meta learner) to explicitly identify the targets of base classes, i.e., the regions that do not need to be segmented. Then, the coarse results output by these two learners in parallel are adaptively integrated to yield precise segmentation prediction. Considering the sensitivity of meta learner, we further introduce an adjustment factor to estimate the scene differences between the input image pairs for facilitating the model ensemble forecasting. The substantial performance gains on PASCAL-5 i and COCO-20 i verify the effectiveness, and surprisingly, our versatile scheme sets a new state-of-the-art even with two plain learners. Moreover, in light of the unique nature of the proposed approach, we also extend it to a more realistic but challenging setting, i.e., generalized FSS, where the pixels of both base and novel classes are required to be determined. The source code is available at github.com/chunbolang/BAM.

Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning

“…HSNet [24] exploits neighborhood consensus to disambiguate semantics by analyzing patterns of local neighborhoods in matching tensors. In addition to the above work, BAM [15] utilizes the segmentation results of the base class to guide the generation of unseen classes, and achieves SOTA results. However, the above methods are all based on backbone freeze, and freezing backbone not only reduces the representational ability of the model, but also does not fit distribution to data better.…”

“…Unlike previous work, in this paper we focus on the prospect of fine-tuning backbone in FSS. Therefore, instead of proposing a new model, we adopt the classic PFENet [34] and BAM [15] as our baselines. Our SVF enables these methods to further improve segmentation results.…”

“…• PFENet [34]: As a classic method in FSS, it proposed the Prior Guided Feature Enrichment Network, which has a huge impact on the subsequent FSS methods. • BAM [15]: As the state-of-the-art method in FSS, it represents the most cutting-edge results in FSS.…”

“…Therefore, different training strategies are NOT the key to the success of SVF. Training Tricks: Following the same setting of BAM [15], we remove some images containing novel classes of the test set from the training set. This is a novel trick in FSS to further improve the performance.…”

“…Test image of COCO-20 i : We found that the number of test sets used in previous work was different when testing on COCO. For example, BAM [15], HSNet [25] were tested with 1000 images, yet Yang [42] was tested with 4000 images, and CyCTR [47] was tested with 5000 images. This is very detrimental to the development of the community.…”