Boosting Few-shot Semantic Segmentation with Transformers

Sun, Guolei; Liu, Yun; Liang, Jingyun; Gool, Luc Van

doi:10.48550/arxiv.2108.02266

Cited by 3 publications

(9 citation statements)

References 49 publications

(87 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Inspired by few-shot learning paradigm [48,57], which aims to learn-to-learn a model for a novel task with only a limited number of samples, fewshot segmentation has received considerable attention. Following the success of [54], prototypical networks [57] and numerous other works [8,25,30,32,43,55,59,68,[75][76][77]82] proposed to utilize a prototype extracted from support samples, which is used to refine the query features to contain the relevant support information. In addition, inspired by [80] that observed the use of high-level features leads to a performance drop, [62] proposed to utilize high-level features by computing a prior map which takes maximum score within a correlation map.…”

Section: Related Workmentioning

confidence: 99%

“…In addition, inspired by [80] that observed the use of high-level features leads to a performance drop, [62] proposed to utilize high-level features by computing a prior map which takes maximum score within a correlation map. Many variants [59,78] extended this idea of utilizing prior maps to guide the feature learning.…”

Section: Related Workmentioning

confidence: 99%

“…As concurrent works, [81] utilizes transformers and proposes to use a cycle-consistent attention mechanism to refine the feature maps to be more discriminative, without considering aggregation of matching scores. [59] propose global and local enhancement module to refine the features using transformers and convolutions in the decoder, respectively. [37] focuses solely on the transformer-based classifier by freezing the encoder and decoder, and adapting only the classifier to the task.…”

Section: Related Workmentioning

confidence: 99%

“…The key to few-shot segmentation is how to effectively utilize provided support samples for a query image. While conventional methods [25,59,62,77,81] attempted to utilize global-or part-level prototypes extracted from support features, recent methods [29,38,67,74,79,81] attempted to leverage pixel-wise relationships between query and support. One of the latest work, HSNet [38], attempts to aggregate the matching scores with 4D convolutions, but it lacks an ability to consider interactions among the matching scores due to the inherent nature of convolutions.…”

Section: Motivation and Overviewmentioning

confidence: 99%

“…Pyramidal Processing. Analogous to [38,59], we also utilize the coarse-to-fine approach through a pyramidal processing as illustrated in Fig. 2.…”

Section: Pyramidal Transformer Encodermentioning

confidence: 99%

See 4 more Smart Citations

Cost Aggregation Is All You Need for Few-Shot Segmentation

Hong¹,

Cho²,

Nam³

et al. 2021

Preprint

View full text Add to dashboard Cite

We introduce a novel cost aggregation network, dubbed Volumetric Aggregation with Transformers (VAT), to tackle the few-shot segmentation task by using both convolutions and transformers to efficiently handle high dimensional correlation maps between query and support. In specific, we propose our encoder consisting of volume embedding module to not only transform the correlation maps into more tractable size but also inject some convolutional inductive bias and volumetric transformer module for the cost aggregation. Our encoder has a pyramidal structure to let the coarser level aggregation to guide the finer level and enforce to learn complementary matching scores. We then feed the output into our affinity-aware decoder along with the projected feature maps for guiding the segmentation process. Combining these components, we conduct experiments to demonstrate the effectiveness of the proposed method, and our method sets a new state-of-the-art for all the standard benchmarks in few-shot segmentation task. Furthermore, we find that the proposed method attains state-of-the-art performance even for the standard benchmarks in semantic correspondence task although not specifically designed for this task. We also provide an extensive ablation study to validate our architectural choices. The trained weights and codes are available at: https: //seokju-cho.github.io/VAT/.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Motivation and Overviewmentioning

confidence: 99%

“…Pyramidal Processing. Analogous to [38,59], we also utilize the coarse-to-fine approach through a pyramidal processing as illustrated in Fig. 2.…”

Section: Pyramidal Transformer Encodermentioning

confidence: 99%

See 3 more Smart Citations

Cost Aggregation Is All You Need for Few-Shot Segmentation

Hong¹,

Cho²,

Nam³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Transductive Meta-Learning with Enhanced Feature Ensemble for Few-shot Semantic Segmentation

Karimi,

POULLIS

2023

Preprint

View full text Add to dashboard Cite

This paper addresses few-shot semantic segmentation and proposes a novel transductive end-to-end method that overcomes three key problems affecting performance. First, we present a novel ensemble of visual features learned from pretrained classification and semantic segmentation networks with the same architecture. Our approach leverages the varying dis-criminative power of these networks, resulting in visual features that capture rich and diverse information at different depths. Secondly, the pretrained semantic segmentation network serves as a base class extractor, which significantly reduces false positives. Thirdly, a transductive meta-learning method addresses the prediction by first learning the relationship between labeled and unlabeled data points with matching support foreground to query features (intra-class similarity) and then applying this knowledge to predict on the unlabeled query image (intra-object similarity), which simultaneously learns propagation and false positive suppression. To evaluate our method, we performed experiments on benchmark datasets, and the results demonstrate significant improvement, with minimal trainable parameters 2.98M. Specifically, using Resnet-101, we achieve state-of-the-art performance for both 1-shot and 5-shot Pascal-5i , as well as on 1-shot and 5-shot COCO-20i .

show abstract

Dense Cross-Query-and-Support Attention Weighted Mask Aggregation for Few-Shot Segmentation

Shi¹,

Wang²,

Lu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Research into Few-shot Semantic Segmentation (FSS) has attracted great attention, with the goal to segment target objects in a query image given only a few annotated support images of the target class. A key to this challenging task is to fully utilize the information in the support images by exploiting fine-grained correlations between the query and support images. However, most existing approaches either compressed the support information into a few class-wise prototypes, or used partial support information (e.g., only foreground) at the pixel level, causing non-negligible information loss. In this paper, we propose Dense pixel-wise Cross-query-and-support Attention weighted Mask Aggregation (DCAMA), where both foreground and background support information are fully exploited via multi-level pixel-wise correlations between paired query and support features. Implemented with the scaled dot-product attention in the Transformer architecture, DCAMA treats every query pixel as a token, computes its similarities with all support pixels, and predicts its segmentation label as an additive aggregation of all the support pixels' labels-weighted by the similarities. Based on the unique formulation of DCAMA, we further propose efficient and effective one-pass inference for n-shot segmentation, where pixels of all support images are collected for the mask aggregation at once. Experiments show that our DCAMA significantly advances the state of the art on standard FSS benchmarks of PASCAL-5 i , COCO-20 i , and FSS-1000, e.g., with 3.1%, 9.7%, and 3.6% absolute improvements in 1-shot mIoU over previous best records. Ablative studies also verify the design DCAMA.

show abstract

Boosting Few-shot Semantic Segmentation with Transformers

Cited by 3 publications

References 49 publications

Cost Aggregation Is All You Need for Few-Shot Segmentation

Cost Aggregation Is All You Need for Few-Shot Segmentation

Transductive Meta-Learning with Enhanced Feature Ensemble for Few-shot Semantic Segmentation

Dense Cross-Query-and-Support Attention Weighted Mask Aggregation for Few-Shot Segmentation

Contact Info

Product

Resources

About