End-to-end Learning of a Fisher Vector Encoding for Part Features in Fine-grained Recognition

Abstract

Part-based approaches for fine-grained recognition do not show the expected performance gain over global methods, although being able to explicitly focus on small details that are relevant for distinguishing highly similar classes. We assume that part-based methods suffer from a missing representation of local features, which is invariant to the order of parts and can handle a varying number of visible parts appropriately. The order of parts is artificial and often only given by ground-truth annotations, whereas viewpoint variations and occlusions result in parts that are not observable. Therefore, we propose integrating a Fisher vector encoding of part features into convolutional neural networks. The parameters for this encoding are estimated jointly with those of the neural network in an end-to-end manner. Our approach improves state-of-the-art accuracies for bird species classification on CUB-200-2011 from 90.40% to 90.95%, on NA-Birds from 89.20% to 90.30%, and on Birdsnap from 84.30% to 86.97%.

Fine-Grained Adversarial Semi-Supervised Learning

“…Recent top performing supervised learning methods have substantially shown that the most successful strategy to FGVC is obtained by identifying, either explicitly or implicitly, the object parts [14,30,52,92]. The central underlying assumption is that fine-grained information resides within the parts.…”

“…Global methods use the input image as a whole and employ different strategies for pre-training [10], augmentation [31,76], or pooling [41,69,70,98] to exploit the parts. In contrast, localization-based methods approaches apply sophisticated detection techniques in order to determine the regions of the parts [17,29,30,84,87,[93][94][95][96]. The two categories have shown comparable results.…”

“…Recent state-of-the-art methods follow different strategies. The work [30] proposes a layer based on Fisher Vector encoding for aggregating part features, [92] proposes a multi-scale object and part attention approach and the work [87] uses an unsupervised learning scheme to localize informative regions. The work in [52] exploits both large external labeled datasets (the JFT [74] private dataset with 300M images) and large architectures (the AmoebaNet architecture [64] with 550M parameters).…”

Fine-Grained Adversarial Semi-supervised Learning

“…Global methods use the input image as a whole and employ different strategies for pre-training [10], augmentation [31,76], or pooling [41,69,70,98] to exploit the parts. In contrast, localization-based methods approaches apply sophisticated detection techniques in order to determine the regions of the parts [17,29,30,84,87,[93][94][95][96]. The two categories have shown comparable results.…”