Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-Grained Image Recognition

Abstract

Learning subtle yet discriminative features (e.g., beak and eyes for a bird) plays a significant role in fine-grained image recognition. Existing attention-based approaches localize and amplify significant parts to learn fine-grained details, which often suffer from a limited number of parts and heavy computational cost. In this paper, we propose to learn such fine-grained features from hundreds of part proposals by Trilinear Attention Sampling Network (TASN) in an efficient teacher-student manner. Specifically, TASN consists of 1) a trilinear attention module, which generates attention maps by modeling the inter-channel relationships, 2) an attention-based sampler which highlights attended parts with high resolution, and 3) a feature distiller, which distills part features into an object-level feature by weight sharing and feature preserving strategies. Extensive experiments verify that TASN yields the best performance under the same settings with the most competitive approaches, in iNaturalist-2017, CUB-Bird, and Stanford-Cars datasets.

Volumetric Transformer Networks

“…In this section, we comprehensively analyze and evaluate VTNs on two tasks: fine-grained image recognition and instance-level image retrieval. First, we analyze the influence of the different components of VTNs compared to existing spatial deformation modeling methods [28,10,48,73] and the impact of combining VTNs with different backbone networks [55,21] and second-order pooling strategies [38,17,9,36,35]. Second, we compare VTNs with the state-of-the-art methods on fine-grained image recognition benchmarks [64,32,40,22].…”

“…Analysis of the VTN components. To validate the different components of our VTNs, we compare them with previous spatial deformation modeling methods, such as STNs [28], deformable convolution (Def-Conv) [10], saliency-based sampler (SSN) [48], and attention-based sampler (ASN) [73] on fine-grained image recognition benchmarks, such as CUB-Birds [64], Stanford-Cars [32], and FGVC-Aircraft [40]. For the comparison to be fair, we apply these methods at the same layer as ours, i.e., the last convolutional layer.…”

“…We also compare VTNs with the state-of-the-art fine-grained image recognition methods, such as RA-CNN [16], MA-CNN [72], DFL-CNN [63], DT-RAM [36], MAMC [58], NTSN [58], DCL [8], and TASN [73]. Since our VTN is designed as a generic drop-in layer that can be combined with existing backbones and pooling strategies, we report the results of VTNs combined with DCL [8], TASN [73], and iSQRT-COV [35], which are the top-performers on this task. As can be seen in Table 3, our method outperforms the state of the art in most cases.…”

“…of which many variants were proposed, using a recurrent formalism [37], polar transformations [12], deformable convolutional kernels [10], and attention based samplers [48,73]. All of these methods apply the same warping field to all the feature channels.…”

“…Comparison of VTN warping fields with those of existing deformation modeling methods[28,48] on examples from CUB-Birds[64]: (a) input images and source coordinates obtained using (b) STNs[28], (c) SSN[48], (d) ASN[73], and (e), (f), (g), and (h) four feature channel samplers in VTNs. Points with the same color in different images are projected to the same point in the canonical pose.…”