Multimodal Classification of Remote Sensing Images: A Review and Future Directions

Abstract

| Earth observation through remote sensing images allows the accurate characterization and identification of materials on the surface from space and airborne platforms. Multiple and heterogeneous image sources can be available for the same geographical region: multispectral, hyperspectral, radar, multitemporal, and multiangular images can today be acquired over a given scene. These sources can be combined/fused to improve classification of the materials on the surface. Even if this type of systems is generally accurate, the field is about to face new challenges: the upcoming constellations of satellite sensors will acquire large amounts of images of different spatial, spectral, angular, and temporal resolutions. In this scenario, multimodal image fusion stands out as the appropriate framework to address these problems. In this paper, we provide a taxonomical view of the field and review the current methodologies for multimodal classification of remote sensing images.We also highlight the most recent advances, which exploit synergies with machine learning and signal processing: sparse methods, kernel-based fusion, Markov modeling, and manifold alignment. Then, we illustrate the different approaches in seven challenging remote sensing applications: 1) multiresolution fusion for multispectral image classification; 2) image downscaling as a form of multitemporal image fusion and multidimensional interpolation among sensors of different spatial, spectral, and temporal resolutions; 3) multiangular image clas, sification; 4) multisensor image fusion exploiting physicallybased feature extractions; 5) multitemporal image classification of land covers in incomplete, inconsistent, and vague image sources; 6) spatiospectral multisensor fusion of optical and radar images for change detection; and 7) cross-sensor adaptation of classifiers. The adoption of these techniques in operational settings will help to monitor our planet from space in the very near future.

Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources

“…Data fusion is one of the fast-moving areas of remote sensing [114][115][116]: due to the recent increases in availability of sensor data, the perspectives of using big and heterogeneous data to study environmental processes have become more tangible. It is a special instance of the more general problem of super-resolution.…”

Multi-resolution classification of urban areas using hierarchical symmetric Markov mesh models

“…The main difficulty is to develop a classifier that jointly utilizes the benefits of multi-band and multi-resolution input data while maintaining a good trade-off between accuracy and computation time. [1]- [4]. Classification techniques as examples of inverse problems solvers, can be regarded as the process that estimates hidden information (or latent variables) x (i.e., urban land cover class labels) from observations y (i.e., satellite data) attached to a set of nodes S. In this framework, Markov random field (MRF) models are widely used classification since they provide a convenient and consistent way of integrating contextual information into the classification scheme [5].…”

Remote Sensing and Cropping Practices: A Review

“…Realizing its potential requires, at least, the development of robust data fusion techniques [211]. However, the joint exploitation of several data sources has led to new data processing issues, such as massive data storage and computationally-intensive processing [212].…”