Knowledge Guided Disambiguation for Large-Scale Scene Classification With Multi-Resolution CNNs

Abstract

Convolutional neural networks (CNNs) have made remarkable progress on scene recognition, partially due to these recent large-scale scene datasets, such as the Places and Places2. Scene categories are often defined by multi-level information, including local objects, global layout, and background environment, thus leading to large intra-class variations. In addition, with the increasing number of scene categories, label ambiguity has become another crucial issue in large-scale classification. This paper focuses on large-scale scene recognition and makes two major contributions to tackle these issues. First, we propose a multi-resolution CNN architecture that captures visual content and structure at multiple levels. The multi-resolution CNNs are composed of coarse resolution CNNs and fine resolution CNNs, which are complementary to each other. Second, we design two knowledge guided disambiguation techniques to deal with the problem of label ambiguity: 1) we exploit the knowledge from the confusion matrix computed on validation data to merge ambiguous classes into a super category and 2) we utilize the knowledge of extra networks to produce a soft label for each image. Then, the super categories or soft labels are employed to guide CNN training on the Places2. We conduct extensive experiments on three large-scale image datasets (ImageNet, Places, and Places2), demonstrating the effectiveness of our approach. Furthermore, our method takes part in two major scene recognition challenges, and achieves the second place at the Places2 challenge in ILSVRC 2015, and the first place at the LSUN challenge in CVPR 2016. Finally, we directly test the learned representations on other scene benchmarks, and obtain the new state-of-the-art results on the MIT Indoor67 (86.7%) and SUN397 (72.0%). We release the code and models at https://github.com/wanglimin/MRCNN-Scene-Recognition.

FOSNet: An End-to-End Trainable Deep Neural Network for Scene Recognition

“…When a mixed CCM-CCG model is used, our FOSNet achieves state-of-the-art accuracy of 60.14% on the Places 2, and it is the first time that the accuracy exceeds 60% on the dataset. [1] 56.2 Gaze Shifting-CNN+SVM [19] 56.2 MetaObject-CNN [15] 58.11 Places365-VGG-SVM [28] 63.24 Three [5] 70.17 Hybrid CNN [21] 70.69 Sparse Representation [23] 71.08 Multi-Resolution CNNs [7] 72.0 CNN-SMN [14] 72.6 PatchNet [22] 73.0 SDO [6] 73.41 Adi-Red [13] 73.59 SOSF+CFA+GAF [20] 78…”

“…In addition, the traits that features appearing in each image region within scene images are all similar was used in [18]. A super category was proposed to solve the problem that the scene categories have label ambiguity in [7]. A deep gaze shifting kernel was developed to distinguish sceneries from different categories in [19].…”

“…For a backbone network, the SE-ResNeXt-101 model, which is a combination of ResNeXt [27] with SE-Network [3], was used for Object-Net and PlacesNet in FOSNet. The standard 10-crop testing method [7] is used for comparison with other methods, and an Adi-Red [13] 41.87 -Places365-VGG [28] 55.24 -CCM [9] 56.82 86.92 CNN-SMN [14] 57.1 -SOSF+CFA+GAF [20] 57.27 -Multi-Resolution CNNs [7] 58.3 87.3 Places2-365-CNN [43] 58.93 88.52 SE-Resnet-152 [3] 59 evaluation measurement is the average classification accuracy of 10 crops.…”

“…RBoW [18] 37.93 DPM+GIST+SP [17] 43.1 Adi-Red [13] 73.59 Gaze Shifting-CNN+SVM [19] 75.1 ResNet-152-DFT + [4] 76.5 Places365-VGG-SVM [28] 76.53 DAG-CNN [1] 77.5 MetaObject-CNN [15] 78.9 VS-CNN [16] 80.37 Hybrid CNN [21] 85.97 Three [5] 86.04 PatchNet [22] 86.2 CNN-SMN [14] 86.5 Multi-Resolution CNNs [7] 86.7 SDO [6] 86.76 Sparse Representation [23] 87.22 SOSF+CFA+GAF [20] 89 computationally very expensive.…”

“…For example, the scene image traits that a scene image consists of a combination of several objects and the objects in the image possesses much information about the category of the scene was used in [5], [6]. The traits that the scene categories labels are inherently ambiguous was used in [7]. The traits of naturalness and openness in scene images were exploited in [8].…”