Capacity and Limits of Multimodal Remote Sensing: Theoretical Aspects and Automatic Information Theory-Based Image Selection

Abstract

Although multimodal remote sensing data analysis can strongly improve the characterization of physical phenomena on Earth's surface, nonidealities and estimation imperfections between records and investigation models can limit its actual information extraction ability. In this paper, we aim at predicting the maximum information extraction that can be reached when analyzing a given dataset. By means of an asymptotic information theory-based approach, we investigate the reliability and accuracy that can be achieved under optimal conditions for multimodal analysis as a function of data statistics and parameters that characterize the multimodal scenario to be addressed. Our approach leads to the definition of two indices that can be easily computed before the actual processing takes place. Moreover, we report in this paper how they can be used for operational use in terms of image selection in order to maximize the robustness of the multimodal analysis, as well as to properly design data collection campaigns for understanding and quantifying physical phenomena. Experimental results show the consistency of our approach.

A graph representation based on fluid diffusion model for data analysis: theoretical aspects and enhanced community detection

“…This approach is very popular within the scientific community, because of its high degree of implementability. However, this does not always reflect in good performance in terms of community detection, especially in operational scenarios [2], [15], [17], [20], [49]. Hence, directly applying these architectures to multimodal data analysis might lead to strong limitations of the multimodal community detection performance.…”

“…From heat to fluid: a new graph representation 1) Motivations of a new graph representation: As previously mentioned, investigating the graph structures induced by the datasets by exploiting the heat propagation analogy in terms of information inference has been proven to be effective and efficient for a wide range of applications and methodological research instances. Nonetheless, these architectures might fail in addressing several data analysis issues that can occur when dealing with multimodal records, especially in operational scenarios [17], [20].…”

“…They would in fact limit the full exploitation of the available training set, either in terms of information extraction or contextaware inference. Moreover, the aforementioned points would reverberate in terms of degradation of confidence and precision of the analysis, as well as restriction of the ability to fully explain and interpret the records under exam [17], [18], [20], [37].…”

“…Finally, the efficiency and the explainability of the data analysis must be addressed, so that near real time applications and thorough interpretation of the data can be carried out. This situation is typically not addressed by the graph signal processing methods in technical literature, as these architectures might not be able to adequately extract information from multimodal datasets, especially in an unsu-pervised fashion [17]- [20].…”

Citywide Estimation of Parking Space Using Aerial Imagery and Osm Data Fusion With Deep Learning and Fine-Grained Annotation

“…In addition to the above mentioned approaches, where the input data comes from a single modality, several studies investigated the fusion of multi-modal and multi-temporal data via neural networks (Chlaily et al, 2020, Hong et al, 2020. Thereby, special attention was paid to the way the data was fused within the networks.…”