Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification From the Bottom Up

Abstract

Given a training dataset composed of images and corresponding category labels, deep convolutional neural networks show a strong ability in mining discriminative parts for image classification. However, deep convolutional neural networks trained with image level labels only tend to focus on the most discriminative parts while missing other object parts, which could provide complementary information. In this paper, we approach this problem from a different perspective. We build complementary parts models in a weakly supervised manner to retrieve information suppressed by dominant object parts detected by convolutional neural networks. Given image level labels only, we first extract rough object instances by performing weakly supervised object detection and instance segmentation using Mask R-CNN and CRF-based segmentation. Then we estimate and search for the best parts model for each object instance under the principle of preserving as much diversity as possible. In the last stage, we build a bi-directional long short-term memory (LSTM) network to fuze and encode the partial information of these complementary parts into a comprehensive feature for image classification. Experimental results indicate that the proposed method not only achieves significant improvement over our baseline models, but also outperforms state-of-the-art algorithms by a large margin (6.7%, 2.8%, 5.2% respectively) on Stanford Dogs 120, Caltech-UCSD Birds 2011-200 and Caltech 256.

Text-Embedded Bilinear Model for Fine-Grained Visual Recognition

“…Nowadays, researchers focus on how to get better categorization accuracy under the weakly-supervised setting. Some works [10,19,48] utilize a localization-classification network and some of them [28,49] learn a discriminative feature representation by end-to-end feature encoding. Different from these models which exploit only visual information, we leverage textual descriptions as well as mine the complementary information between texts and images, which make our proposed approach learn more discriminative attribute-related features and obtain a 3.7% higher accuracy than the best performing result.…”

“…√ indicates YES and blank expresses NO. 82.8 Mask-CNN [44] √ 87.3 STN [19] 84.1 Bilinear-CNN [28] 84.1 Low-rank Bilinear [21] 84.2 RA-CNN [8] 85.3 HBP [49] 87.1 DFL-CNN [42] 87.4 NTS-Net [48] 87.5 DCL [6] 87.8 TASN [56] 87.9 WS-BAN [15] 88.8 Ge et al [10] 90.4 Triplet-A [7] √ Manual labour 80.7 Multi-grained [ [34] 89.5 SDR [1] 90.5 ResNet-50 [11] 92.4 RIIR [46] 94.0 NAC-CNN [38] 95.3 MGE-CNN [51] 95.9 PBC [16] 96.1 CVL [12] 96.2 TEB (ours) 96.6…”

“…Peng et al [32] proposed a weakly-supervised formulation for fine-grained visual recognition to make sure that the model could select discriminative regions, and a specialized clustering algorithm is used to integrate the features of these regions. Ge et al [10] built complementary parts models in a weakly supervised manner to retrieve information via object detection and instance segmentation techniques. Moreover, visual attention models [5,14,45] have been intensively proposed to automatically search the informative regions, and some works [8,39,48,55,56] also apply this technique to fine-grained recognition task.…”

Exploiting Category Similarity-Based Distributed Labeling for Fine-Grained Visual Classification

“…Performances of baseline methods are directly from Maximum-Entropy [32]. We also compared our proposed CSDL with state-of-the-art weakly supervised fine-grained methods, including BCNN [26], Kernel Pooling [30], MA-CNN [20], NTS-Net [24], DFL-CNN [25], iSQRT-COV [31], Part Model [56], TASN [39], DCL [40]. Performances of existing methods are all taken directly from their original papers.…”

Visual Sentiment Analysis With Active Learning

“…For image classification task, Deep Neural Network (DNN) has demonstrated the ability of learning representative features by many literatures [4]. It can get better classification accuracy when using more annotated samples.…”