Pixel-Wise Classification of High-Resolution Ground-Based Urban Hyperspectral Images with Convolutional Neural Networks

Abstract

Using ground-based, remote hyperspectral images from 0.4–1.0 micron in ∼850 spectral channels—acquired with the Urban Observatory facility in New York City—we evaluate the use of one-dimensional Convolutional Neural Networks (CNNs) for pixel-level classification and segmentation of built and natural materials in urban environments. We find that a multi-class model trained on hand-labeled pixels containing Sky, Clouds, Vegetation, Water, Building facades, Windows, Roads, Cars, and Metal structures yields an accuracy of 90–97% for three different scenes. We assess the transferability of this model by training on one scene and testing to another with significantly different illumination conditions and/or different content. This results in a significant (∼45%) decrease in the model precision and recall as does training on all scenes at once and testing on the individual scenes. These results suggest that while CNNs are powerful tools for pixel-level classification of very high-resolution spectral data of urban environments, retraining between scenes may be necessary. Furthermore, we test the dependence of the model on several instrument- and data-specific parameters including reduced spectral resolution (down to 15 spectral channels) and number of available training instances. The results are strongly class-dependent; however, we find that the classification of natural materials is particularly robust, especially the Vegetation class with a precision and recall >94% for all scenes and model transfers and >90% with only a single training instance.

Atmospheric correction of vegetation reflectance with simulation-trained deep learning for ground-based hyperspectral remote sensing

“…5 are trained, validated, and tested using vegetation spectral reflectance of systematically reduced spectral resolution. CNN models applied to spectral analysis are known to be sensitive to the filter sizes in their convolutional layers [ 72 ]. Therefore, in each model the number of filters and their kernel sizes were reduced in proportion to the reduction in spectral resolution in order to maintain an equivalent spectral filter across models.…”

Smart and Sentient Retail High Streets

“…Hyperspectral imaging is capable of detecting signatures across different wavelengths of light, while multispectral can peer within wavelengths. There are now a wide variety of applications of both to the study of streetscapes in general terms, as sites for the presence of environmental features that can be detected in different wavelengths [102][103][104][105][106]. In other examples, the sensing is targeted specifically at pedestrians on streets.…”

Accuracy Assessment in Convolutional Neural Network-Based Deep Learning Remote Sensing Studies—Part 1: Literature Review

“…A common approach in RS DL studies is to average individual metrics over all the mapped classes. Examples of such metrics include mean recall (sometimes given the term mean accuracy [118], which for binary data is equivalent to balanced accuracy [101]); mean precision [111,119] (not to be confused with average precision, see Section 4.3.2. ); mean F1 [120], and mean IoU (mIOU) [121].…”