Self-supervised audiovisual representation learning for remote sensing data

Heidler, Konrad; Mou, Lichao; Hu, Di; Pu, Jun; Li, Guangyao; Gan, Chuang; Wen, Ji-Rong; Zhu, Xiao Xiang

doi:10.1016/j.jag.2022.103130

Cited by 18 publications

(12 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The primary purpose of data fusion in multi-view (MV) learning is to combine the information from different perspectives (views) to provide a broader understanding of the phenomena and improve the predictive performance of machine learning models [28]. However, sometimes the goal could be just to get an embedding to search for similar views, as is the case of MV alignment or representation learning [8], [29], [30]. This alignment is the base of contrastive learning [27], where a model is used to project the data into a shared subspace for each view.…”

Section: Literature Reviewmentioning

confidence: 99%

Common Practices and Taxonomy in Deep Multiview Fusion for Remote Sensing Applications

Mena,

Arenas,

Nuske

et al. 2024

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

The advances in remote sensing technologies have boosted applications for Earth observation. These technologies provide multiple observations or views with different levels of information. They might contain static or temporary views with different levels of resolution, in addition to having different types and amounts of noise due to sensor calibration or deterioration. A great variety of deep learning models have been applied to fuse the information from these multiple views, known as deep multiview or multi-modal fusion learning. However, the approaches in the literature vary greatly since different terminology is used to refer to similar concepts or different illustrations are given to similar techniques. This article gathers works on multi-view fusion for Earth observation by focusing on the common practices and approaches used in the literature. We summarize and structure insights from several different publications concentrating on unifying points and ideas. In this manuscript, we provide a harmonized terminology while at the same time mentioning the various alternative terms that are used in literature. The topics covered by the works reviewed focus on supervised learning with the use of neural network models. We hope this review, with a long list of recent references, can support future research and lead to a unified advance in the area.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Common Practices and Taxonomy in Deep Multiview Fusion for Remote Sensing Applications

Mena,

Arenas,

Nuske

et al. 2024

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

show abstract

“…Here, we compare the results of four schemes, ADVANCE [66], SoundingEarth [68], attention + CNN, and AiT, to demonstrate the effectiveness of the schemes based only on Transformer. Among them, the acoustic characteristics of the four schemes are common (use the log-Mel spectrogram).…”

Section: Audio Experimentsmentioning

confidence: 99%

“…Here, we compare the results of ADVANCE [66], SoundingEarth [68], and our twostage hybrid fusion strategy and show the quantitative results of different fusion strategies on three metrics (precision, recall, F1-score). The ADVANCE scheme transfers sound event knowledge to aerial scene recognition tasks to improve scene recognition performance.…”

Section: Audiovisual Experimentsmentioning

confidence: 99%

Two-Stage Fusion-Based Audiovisual Remote Sensing Scene Classification

Wang,

Liu,

Huang

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

Scene classification in remote sensing is a pivotal research area, traditionally relying on visual information from aerial images for labeling. The introduction of ground environment audio as a novel geospatial data source adds valuable information for scene classification. However, bridging the structural gap between aerial images and ground environment audio is challenging, rendering popular two-branch networks ineffective for direct data fusion. To address this issue, the study in this research presents the Two-stage Fusion-based Audiovisual Classification Network (TFAVCNet). TFAVCNet leverages both audio and visual modules to extract deep semantic features from ground environmental audio and remote sensing images, respectively. The audiovisual fusion module combines and fuses information from both modalities at the feature and decision levels, facilitating joint training and yielding a more-robust solution. The proposed method outperforms existing approaches, as demonstrated by the experimental results on the ADVANCE dataset for remote sensing audiovisual scene classification, offering an innovative approach to enhanced scene classification.

show abstract

“…Self-supervised learning approaches in earth observation focus on learning representation from unlabeled data as pretraining following contrastive [26]- [30] and generative [31]- [33] approaches. Learned representations can be used for downstream tasks supported by fine-tuning with curated labels in scene classification [26], [27], [29], [31], [32] and semantic segmentation [28]- [30] tasks. This two-step strategy is also leaning toward semi-supervision and knowledge distillation [34].…”

Section: Introductionmentioning

confidence: 99%

Self-Supervised Variational Autoencoder for Unsupervised Object Counting from Very-High-Resolution Satellite Imagery: Applications in Dwelling Extraction in FDP Settlement Areas

Gella,

Gangloff,

Wendt

et al. 2024

IEEE Trans. Geosci. Remote Sensing

View full text Add to dashboard Cite

In supervised learning, deep learning models demand a large corpus of annotated data for object detection and classification tasks. This constrains their utility in humanitarian emergency response. To overcome this problem, we have proposed an unsupervised dwelling counting from very high-resolution satellite imagery by combining a Variational Autoencoder(VAE) with an anomaly detection approach. When VAEs are applied in earth observation for dwelling localization and counting, we observed two critical limitations (1) the balance between reconstruction and good latent code, where in-favour of good reconstruction of dwellings leads to weak anomaly score maps that fail to properly localize dwellings (2) limited spatiotemporal invariance of the learned latent code. When the model is trained with datasets obtained from different geography and time, it fails to properly localize dwellings. For the first problem, we introduced self-supervision by creating synthetic anomalies. For the second problem, we introduced latent space conditioning. The approach is tested on 9 very high-resolution images obtained from six Forcibly Displaced People settlement areas. Results indicate that combining VAE with an anomaly detection approach has reached an AUC value ranging from 0.70 at complex settlements towards 0.98 at relatively less complex settlement areas. Similarly, an MAE value of 56.67 towards 5.03 is achieved for dwelling counting. Joint training of combined datasets with latent space conditioning and self-supervision enabled the achievement of results better than classical VAE, with improved spatiotemporal transferability of the model with more crisp and strong anomaly maps. Overall implementation code will be available at https://github.com/getch-geohum/SSL-VAE.

show abstract

Self-supervised audiovisual representation learning for remote sensing data

Cited by 18 publications

References 53 publications

Common Practices and Taxonomy in Deep Multiview Fusion for Remote Sensing Applications

Common Practices and Taxonomy in Deep Multiview Fusion for Remote Sensing Applications

Two-Stage Fusion-Based Audiovisual Remote Sensing Scene Classification

Self-Supervised Variational Autoencoder for Unsupervised Object Counting from Very-High-Resolution Satellite Imagery: Applications in Dwelling Extraction in FDP Settlement Areas

Contact Info

Product

Resources

About