Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning

He, Yangji; Liang, Weihan; Zhao, Dongyang; Zhou, Hongyu; Ge, Weifeng; Yu, Yizhou; Zhang, Wenqiang

doi:10.1109/cvpr52688.2022.00891

Cited by 32 publications

(7 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some researchers utilize long-range relationships of local image patches and transformer networks with the selfattention mechanism to achieve better image recognition performance. Recent works [6] [7] [8] have proposed either combining or replacing CNNs with transformer networks. These efforts often entail employing long token sequence learning within transformers to mitigate catastrophic forgetting.…”

Section: Introductionmentioning

confidence: 99%

ViTFSL-Baseline: A Simple Baseline of Vision Transformer Network for Few-Shot Image Classification

Wang,

Pan

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Few-shot image classification, whose goal is to generalize to unseen tasks with scarce labeled data, has developed rapidly over the years. However, in traditional few-shot learning methods with CNNs, non-local features and long-rang dependencies of the image may be lost, and this leads to a poor generalization of the trained model. With the advantage of the self-attention mechanism of Transformer, researchers have tried to use vision transformer to improve few-shot learning recently. However, these methods are more complicated and take up a lot of computing resources, and there is no baseline to measure their effectiveness. We propose a new method called ViTFSL-baseline. We take advantage of vision transformer and train our model on all train set without episodic training. Meanwhile, we design a new nearest-neighbor classifier NCL to used for few-shot image classification. Furthermore, in order to narrow the gap between difference of same class, we introduce centroid calibration in classifier after the feature extraction of backbone. We run the experiments on popular benchmarks to show that our method is a simple and effective for few-shot image classification. Our approach could be taken as the baseline upon vision transformer for few-shot learning.

show abstract

Section: Introductionmentioning

confidence: 99%

ViTFSL-Baseline: A Simple Baseline of Vision Transformer Network for Few-Shot Image Classification

Wang,

Pan

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Moreover, token sparsification can also remove spurious correlations to improve the testing accuracy (Likhomanenko et al, 2021;Zhu et al, 2021a). This insight provides a guideline in designing token sparsification and few-shot learning methods for Transformer (He et al, 2022;Guibas et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity

Hong-kang¹,

Wang²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Vision Transformers (ViTs) with self-attention modules have recently achieved great empirical success in many vision tasks. Due to non-convex interactions across layers, however, the theoretical learning and generalization analysis is mostly elusive. Based on a data model characterizing both label-relevant and label-irrelevant tokens, this paper provides the first theoretical analysis of training a shallow ViT, i.e., one self-attention layer followed by a two-layer perceptron, for a classification task. We characterize the sample complexity to achieve a zero generalization error. Our sample complexity bound is positively correlated with the inverse of the fraction of label-relevant tokens, the token noise level, and the initial model error. We also prove that a training process using stochastic gradient descent (SGD) leads to a sparse attention map, which is a formal verification of the general intuition about the success of attention. Moreover, this paper indicates that a proper token sparsification can improve the test performance by removing label-irrelevant and/or noisy tokens, including spurious correlations. Empirical experiments on synthetic data and CIFAR-10 dataset justify our theoretical results and generalize to deeper ViTs.

show abstract

“…In addition, transformer-based methods often need significant training iterations and are hard to converge during the training process. [103] proposed hierarchically cascaded transformers that exploit spectral tokens pooling and surrogates learning to improve data efficiency. However, it used multiple transformers and caused the instability of training.…”

Section: Few-shot Classification Methodsmentioning

confidence: 99%

“…In this section, we evaluate our method on popular benchmark datasets for few-shot classifi- to 80 epochs, while previous methods [103,34] needs more than 400 epochs to converge.…”

Section: Implementation Detailsmentioning

confidence: 99%

“…In addition, we analyze the abilities of the core components in ablation studies and present the training details of the ViT.Datasets. Following the evaluation procedure from previous works[93,103], we perform experiments on three popular benchmark datasets for few-shot classification, including miniImageNet, tieredImageNet, and CIFAR-FS. miniImageNet is a subset of Ima-geNet[146], consisting of 60,000 images that are distributed over100 classes.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Reliable and structural deep neural networks

Yuan¹

View full text Add to dashboard Cite

Deep neural networks have dominated a wide range of computer vision research recently. However, recent studies have shown that deep neural networks are sensitive to adversarial perturbations. The limitations of deep networks cause reliability concerns in real-world problems and demonstrate that computational behaviors differ from humans. In this dissertation, we focus on investigating the characteristic of deep neural networks. The first part of this dissertation proposed an effective defense method against adversarial examples. We introduced an ensemble generative network with feedback loops, which use the feature-level denoising modules to improve the defense capability for adversarial examples. We then discussed the vulnerability of deep neural networks. We explored a consistency and sensitivity-guided attack method in a low-dimensional space, which can effectively generate adversarial examples, even in a black-box manner. Our proposed approach illustrated that the adversarial examples are transferable across different networks and universal in deep networks. The last part of this dissertation focuses on rethinking the structure and behavior of deep neural networks. Rather than enhancing defense methods against attacks, we take a further step toward developing a new structure of neural networks, which provide a dynamic link between the feature map representation and their graph-based structural representation. In addition, we introduced a new feature interaction method based on the vision transformer. The new structure can learn to dynamically select the most discriminative features and help deep networks improve the generalization ability.

show abstract

Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning

Cited by 32 publications

References 35 publications

ViTFSL-Baseline: A Simple Baseline of Vision Transformer Network for Few-Shot Image Classification

ViTFSL-Baseline: A Simple Baseline of Vision Transformer Network for Few-Shot Image Classification

A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity

Reliable and structural deep neural networks

Contact Info

Product

Resources

About