Transformer with Peak Suppression and Knowledge Guidance for Fine-grained Image Recognition

Abstract

Fine-grained image recognition is challenging because discriminative clues are usually fragmented, whether from a single image or multiple images. Despite their significant improvements, the majority of existing methods still focus on the most discriminative parts from a single image, ignoring informative details in other regions and lacking consideration of clues from other associated images. In this paper, we analyze the difficulties of fine-grained image recognition from a new perspective and propose a transformer architecture with the peak suppression module and knowledge guidance module, which respects the diversification of discriminative features in a single image and the aggregation of discriminative clues among multiple images. Specifically, the peak suppression module first utilizes a linear projection to convert the input image into sequential tokens. It then blocks the token based on the attention response generated by the transformer encoder. This module penalizes the attention to the most discriminative parts in the feature learning process, therefore, enhancing the information exploitation of the neglected regions. The knowledge guidance module compares the image-based representation generated from the peak suppression module with the learnable knowledge embedding set to obtain the knowledge response coefficients. Afterwards, it formalizes the knowledge learning as a classification problem using response coefficients as the classification scores. Knowledge embeddings and image-based representations are updated during training simultaneously so that the knowledge embedding includes a large number of discriminative clues for different images of the same category. Finally, we incorporate the acquired knowledge embeddings into the image-based representations as comprehensive representations, leading to significantly higher recognition performance. Extensive evaluations on the six popular datasets demonstrate the advantage of the proposed method in performance.

ViT-FOD: A Vision Transformer based Fine-grained Object Discriminator

“…For example, TransFG [16] proposes the first ViT-based fine-grained classification method, in which a part selection module is designed to select discriminative tokens. Following that, several methods [19,29,39] also apply ViT to FGVC. Thereinto, TPSKG [29] proposes a peak suppression module which penalizes the attention to the most discriminative part and a knowledge guidance module to obtain the knowledge response coefficients.…”

“…Following that, several methods [19,29,39] also apply ViT to FGVC. Thereinto, TPSKG [29] proposes a peak suppression module which penalizes the attention to the most discriminative part and a knowledge guidance module to obtain the knowledge response coefficients. RAMS-Trans [19] learns discriminative region attention in a multi-scale way.…”

“…ViT [7] ViT-B_16 89.8 RAMS-Trans [19] ViT-B_16 91.3 TPSKG [29] ViT-B_16 91.3 FFVT [39] ViT-B_16 91.6 TransFG [16] ViT-B_16 91.7…”

“…Inspired by this, Vision Transformer (ViT) [7] with multiple self-attention layers has also been introduced into Computer Vision and attracted extensive attention. More recently, several works [16,19,29,39] have tried to apply ViT to FGVC and make a breakthrough. These primary ViT-based attempts have significantly surpassed existing convolution-based methods, demonstrating the superiority of ViT in FGVC.…”

Dynamic MLP for Fine-Grained Image Classification by Leveraging Geographical and Temporal Information

“…Other researches focus on extracting more useful features from multi-channel networks [5,57] or contrastive learning [1,13]. TransFG [17] and TPSKG [27] have recently used the Transformer architecture to improve classification performance. The fine-grained methods generally suffer from a complex pipeline and enormous manual design.…”