Integrating Multilayer Features of Convolutional Neural Networks for Remote Sensing Scene Classification

Li, Erzhu; Xia, Junshi; Du, Peijun; Lin, Cong; Samat, Alim

doi:10.1109/tgrs.2017.2711275

Cited by 276 publications

(179 citation statements)

References 51 publications

Supporting

Mentioning

179

Contrasting

Order By: Relevance

“…In [56,72,74], deep features and hand-crafted features were combined to get a discriminative scene presentation. In [54,55], multilayer features based on a convolutional neural network were fused to get better results. In addition, the work reported in [75,76] attempted to adopt a multiscale feature fusion strategy.…”

Section: Comparisons With the Most Recent Methodsmentioning

confidence: 99%

Combing Triple-Part Features of Convolutional Neural Networks for Scene Classification in Remote Sensing

Hong

2019

Remote Sensing

View full text Add to dashboard Cite

High spatial resolution remote sensing (HSRRS) images contain complex geometrical structures and spatial patterns, and thus HSRRS scene classification has become a significant challenge in the remote sensing community. In recent years, convolutional neural network (CNN)-based methods have attracted tremendous attention and obtained excellent performance in scene classification. However, traditional CNN-based methods focus on processing original red-green-blue (RGB) image-based features or CNN-based single-layer features to achieve the scene representation, and ignore that texture images or each layer of CNNs contain discriminating information. To address the above-mentioned drawbacks, a CaffeNet-based method termed CTFCNN is proposed to effectively explore the discriminating ability of a pre-trained CNN in this paper. At first, the pretrained CNN model is employed as a feature extractor to obtain convolutional features from multiple layers, fully connected (FC) features, and local binary pattern (LBP)-based FC features. Then, a new improved bag-of-view-word (iBoVW) coding method is developed to represent the discriminating information from each convolutional layer. Finally, weighted concatenation is employed to combine different features for classification. Experiments on the UC-Merced dataset and Aerial Image Dataset (AID) demonstrate that the proposed CTFCNN method performs significantly better than some state-of-the-art methods, and the overall accuracy can reach 98.44% and 94.91%, respectively. This indicates that the proposed framework can provide a discriminating description for HSRRS images. 2 of 23 color features, spectral features, and multi-feature fusion [23,24]. However, these hand-crafted features are limited in describing complex scenes of HSRRS, which will affect the classification performance. Compared with the low-level methods, the mid-level methods aim to obtain a global representation of a scene by encoding local descriptors, e.g., scale-invariant feature transform, histogram of oriented gradient and color histogram. The bag-of-view-word (BoVW) model is one of the most popular feature encoding approaches [25]. Due to its simplicity and efficiency, the BoVW model is widely applied for mid-level scene description [26][27][28][29]. However, the quantization error of the BoVW method is large, and some important information may be lost. Therefore, many feature coding methods were developed to reduce the reconstruction error, including improved Fisher kernel (IFK) [30], vectors of locally aggregated descriptors (VLAD) [31], spatial pyramid matching kernel (SPM) [32], locality-constrained linear coding (LLC) [33], latent semantic analysis (LSA), probabilistic latent semantic analysis (pLSA) [34,35], and latent dirichlet allocation (LDA) [35]. However, both low-level and mid-level methods are mainly based on hand-crafted features, which are difficult to effectively describe in HSRRS scene images with complex land-cover/land-use (LULC) situations.In recent years, deep-learning-based methods have made a...

show abstract

Section: Comparisons With the Most Recent Methodsmentioning

confidence: 99%

Combing Triple-Part Features of Convolutional Neural Networks for Scene Classification in Remote Sensing

Hong

2019

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…There is an extensive literature on algorithms for geospatial land image processing -often called remote sensing image processing. This broad and active field of research has many branches, such as semantic segmentation [6], [7], target localization [8], and region classification [2], [3], [9], [10], [11]. In the domain of this work, there are different publicly available datasets for land use/land cover classification that can be used as performance benchmarks.…”

Section: Related Workmentioning

confidence: 99%

Hydra: An Ensemble of Convolutional Neural Networks for Geospatial Land Classification

Minetto

Segundo

Sarkar

2019

IEEE Trans. Geosci. Remote Sensing

127

View full text Add to dashboard Cite

2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Abstract-We describe in this paper Hydra, an ensemble of convolutional neural networks (CNN) for geospatial land classification. The idea behind Hydra is to create an initial CNN that is coarsely optimized but provides a good starting pointing for further optimization, which will serve as the Hydra's body. Then, the obtained weights are finetuned multiple times with different augmentation techniques, crop styles, and classes weights to form an ensemble of CNNs that represent the Hydra's heads. By doing so, we prompt convergence to different endpoints, which is a desirable aspect for ensembles. With this framework, we were able to reduce the training time while maintaining the classification performance of the ensemble. We created ensembles for our experiments using two state-of-the-art CNN architectures, ResNet and DenseNet. We have demonstrated the application of our Hydra framework in two datasets, FMOW and NWPU-RESISC45, achieving results comparable to the state-of-the-art for the former and the best reported performance so far for the latter. Code and CNN models are available at https://github.com/maups/hydra-fmow.Index Terms-Geospatial land classification, remote sensing image classification, functional map of world, ensemble learning, on-line data augmentation, convolutional neural network.

show abstract

“…Because of the GPU memory limitations, high-resolution imagery must be segmented into patches for Convolutional Neural Networks (CNN) models, and the label is always attached to a remote sensing image sample [8,10,[33][34][35] in scene classifications. To manage and retrieve patches easily, the previous research studies modified the structure of traditional deep convolution neural networks into two different forms, i.e., cascade and parallel models, according to the characteristics of remote sensing images [36].…”

Section: Scene Classificationmentioning

confidence: 99%

“…For example, a global average pooling layer is used to replace the fully connected network as the classifier [10] or to insert a region-based cascade pooling (RBCP) method between the last normal down-sampling layer and the classifier to aggregate convolutional features from both the pre-trained and the fine-tuned convolutional neural networks [36]. Parallel models try to extract more abundant features for scene classification by designing parallel network structures [35]. All of the above methods achieved satisfactory results.…”

Section: Scene Classificationmentioning

confidence: 99%

Multi-Scale Semantic Segmentation and Spatial Relationship Recognition of Remote Sensing Images Based on an Attention Model

Cui

Wang

et al. 2019

Remote Sensing

View full text Add to dashboard Cite

A comprehensive interpretation of remote sensing images involves not only remote sensing object recognition but also the recognition of spatial relations between objects. Especially in the case of different objects with the same spectrum, the spatial relationship can help interpret remote sensing objects more accurately. Compared with traditional remote sensing object recognition methods, deep learning has the advantages of high accuracy and strong generalizability regarding scene classification and semantic segmentation. However, it is difficult to simultaneously recognize remote sensing objects and their spatial relationship from end-to-end only relying on present deep learning networks. To address this problem, we propose a multi-scale remote sensing image interpretation network, called the MSRIN. The architecture of the MSRIN is a parallel deep neural network based on a fully convolutional network (FCN), a U-Net, and a long short-term memory network (LSTM). The MSRIN recognizes remote sensing objects and their spatial relationship through three processes. First, the MSRIN defines a multi-scale remote sensing image caption strategy and simultaneously segments the same image using the FCN and U-Net on different spatial scales so that a two-scale hierarchy is formed. The output of the FCN and U-Net are masked to obtain the location and boundaries of remote sensing objects. Second, using an attention-based LSTM, the remote sensing image captions include the remote sensing objects (nouns) and their spatial relationships described with natural language. Finally, we designed a remote sensing object recognition and correction mechanism to build the relationship between nouns in captions and object mask graphs using an attention weight matrix to transfer the spatial relationship from captions to objects mask graphs. In other words, the MSRIN simultaneously realizes the semantic segmentation of the remote sensing objects and their spatial relationship identification end-to-end. Experimental results demonstrated that the matching rate between samples and the mask graph increased by 67.37 percentage points, and the matching rate between nouns and the mask graph increased by 41.78 percentage points compared to before correction. The proposed MSRIN has achieved remarkable results. the existing studies do not address the interpretation of spatial relationships between remote sensing objects, which limits the understanding of remote sensing objects, especially when the phenomenon of different objects with the same spectrum in remote sensing appears.The phenomenon of different objects with the same spectrum in remote sensing is quite common. It is difficult to identify objects only by their own textures, spectra, and shape information. Object identification requires multi-scale semantic information and spatially adjacent objects to assist in decision-making. The spatial relationship between remote sensing objects is of great significance to the recognition of remote sensing objects when different objects have the same spectru...

show abstract

Integrating Multilayer Features of Convolutional Neural Networks for Remote Sensing Scene Classification

Cited by 276 publications

References 51 publications

Combing Triple-Part Features of Convolutional Neural Networks for Scene Classification in Remote Sensing

Combing Triple-Part Features of Convolutional Neural Networks for Scene Classification in Remote Sensing

Hydra: An Ensemble of Convolutional Neural Networks for Geospatial Land Classification

Multi-Scale Semantic Segmentation and Spatial Relationship Recognition of Remote Sensing Images Based on an Attention Model

Contact Info

Product

Resources

About