Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition

Abstract

Attention-based learning for fine-grained image recognition remains a challenging task, where most of the existing methods treat each object part in isolation, while neglecting the correlations among them. In addition, the multi-stage or multi-scale mechanisms involved make the existing methods less efficient and hard to be trained end-to-end. In this paper, we propose a novel attention-based convolutional neural network (CNN) which regulates multiple object parts among different input images. Our method first learns multiple attention region features of each input image through the one-squeeze multi-excitation (OSME) module, and then apply the multi-attention multi-class constraint (MAMC) in a metric learning framework. For each anchor feature, the MAMC functions by pulling same-attention same-class features closer, while pushing different-attention or different-class features away. Our method can be easily trained end-to-end, and is highly efficient which requires only one training stage. Moreover, we introduce Dogs-in-the-Wild, a comprehensive dog species dataset that surpasses similar existing datasets by category coverage, data volume and annotation quality. This dataset will be released upon acceptance to facilitate the research of fine-grained image recognition. Extensive experiments are conducted to show the substantial improvements of our method on four benchmark datasets.

Cross-X Learning for Fine-Grained Visual Categorization

“…We begin by briefly reviewing the one-squeeze multiexcitation (OSME) block [28] that learns multiple attention region features for each input image. Let U = [u 1 , · · · , u C ] ∈ R W ×H×C denote the output feature map of Figure 1.…”

“…Although OSME can generate attention-specific features, guiding these features to have semantic meanings is challenging. [28] tackles this by optimizing a metric learning loss which pulls features from the same excitation closer and pushes features from different excitations away. However, optimizing such a loss still poses a challenge and involves a non-trivial sample selection procedure [33].…”

“…Instead of optimizing a metric loss as in [28], we propose to learn semantic features by exploring the correlations between feature maps from different images and different excitation modules. Ideally, we want the extracted features from the same excitation module to have the same semantic meaning, even though they come from different images with different class labels.…”

“…Surprisingly, the performance of our reimplementation of SENet/ResNet-50 surpasses many previous methods. Even though they can improve their performance by employing more advanced architectures, e.g., MAMC-CNN [28] with ResNet-101 (85.2%) and MaxEnt-CNN [4] with DenseNet-161 (83.6%) as reported in their papers, still falling behind us. However, our Cross-X learning can beat ResNet-50 a bit and achieve the state-of-theart performance by combining with SENet-50 and ResNet-50, respectively.…”

“…We propose Cross-X learning, a simple but effective approach that leverages the relationships between different images and between different network layers for robust finegrained recognition. Similar to [28], our approach first generates attention region features via multiple excitation modules, but it further involves two novel components: a crosscategory cross-semantic regularizer (C 3 S) and a cross-layer regularizer (CL). C 3 S is introduced to guide the attention features from different excitation modules to represent different semantic parts.…”