Semantic segmentation of multisensor remote sensing imagery with deep ConvNets and higher-order conditional random fields

Abstract

Aerial images acquired by multiple sensors provide comprehensive and diverse information of materials and objects within a surveyed area. The current use of pretrained deep convolutional neural networks (DCNNs) is usually constrained to three-band images (i.e., RGB) obtained from a single optical sensor. Additional spectral bands from a multiple sensor setup introduce challenges for the use of DCNN. We fuse the RGB feature information obtained from a deep learning framework with light detection and ranging (LiDAR) features to obtain semantic labeling. Specifically, we propose a decision-level multisensor fusion technique for semantic labeling of the very-high-resolution optical imagery and LiDAR data. Our approach first obtains initial probabilistic predictions from two different sources: one from a pretrained neural network fine-tuned on a three-band optical image, and another from a probabilistic classifier trained on LiDAR data. These two predictions are then combined as the unary potential using a higher-order conditional random field (CRF) framework, which resolves fusion ambiguities by exploiting the spatial-contextual information. We utilize graph cut to efficiently infer the final semantic labeling for our proposed higher-order CRF framework. Experiments performed on three benchmarking multisensor datasets demonstrate the performance advantages of our proposed method. © The Authors. Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

Post-Disaster Building Database Updating Using Automated Deep Learning: An Integration of Pre-Disaster OpenStreetMap and Multi-Temporal Satellite Data

“…In addition, since the OSM building map did not provide a precise building mask, particularly for the building boundaries, we verified inaccuracies in some parts of the images. This problem was alleviated by implementing a Conditional Random Field method (CRF), which has been primarily investigated in the literature as a refinement over the U-net or Fully Connected Networks (FCNs) results [56,77,78]. Accordingly, a fully/dense CRF model developed by Krähenbühl and Koltun [79] was employed to optimize the ResUnet results.…”

Dense Dilated Convolutions’ Merging Network for Land Cover Classification

“…Our applied focus in this paper is land cover classification based on remote sensing. Lately, the FCNs and encoderdecoder architectures have been widely adapted and applied to the ISPRS [8] Semantic Labeling Contest [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], and the DeepGlobe CVPR-2018 [9] challenge of automatic classification of land cover types [30], [31], [32], [33], [34], [35], [36]. Paisitkriangkrai et al [20] proposed a scheme for high-resolution land cover classification using a combination of a patchbased CNN and a random forest classification that is trained on hand-crafted features.…”

Improved Winter Wheat Spatial Distribution Extraction Using A Convolutional Neural Network and Partly Connected Conditional Random Field

“…As CRFs have the ability to capture both local and long-range dependencies within an image, they significantly improve CNN segmentation results [59]. The existing CRFs, such as the fully connected CRF modeling processes, are complicated and require a large number of calculations [60]. To complete the calculations, previous studies have used approximate calculations [60,61], reduction of the number of samples involved in modeling [62,63], and introduced conditional independence [64][65][66].…”