Cross-Modal Contrastive Learning for Remote Sensing Image Classification

Feng, Zhixi; Song, Liangliang; Yang, Shuyuan; Zhang, Xinyu; Jiao, Licheng

doi:10.1109/tgrs.2023.3296703

Cited by 11 publications

(1 citation statement)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also, Contrastive-based methods are beginning to be used in the multi-modal remote sensing image classification. Feng et al [39] use joint intra-and cross-modal contrastive learning to mine multi-modal feature representations during pre-training. They design a simple yet effective hybrid cross-modal fusion module in the fine-tuning stage to better compactly integrate complementary information across these modalities for ground object classification.…”

Section: B Self-supervised Learningmentioning

confidence: 99%

SS-MAE: Spatial–Spectral Masked Autoencoder for Multisource Remote Sensing Image Classification

Lin,

Gao,

Shi

et al. 2023

IEEE Trans. Geosci. Remote Sensing

View full text Add to dashboard Cite

Masked image modeling (MIM) is a highly popular and effective self-supervised learning method for image understanding. Existing MIM-based methods mostly focus on spatial feature modeling, neglecting spectral feature modeling. Meanwhile, existing MIM-based methods use Transformer for feature extraction, some local or high-frequency information may get lost. To this end, we propose a spatial-spectral masked auto-encoder (SS-MAE) for HSI and LiDAR/SAR data joint classification. Specifically, SS-MAE consists of a spatial-wise branch and a spectral-wise branch. The spatial-wise branch masks random patches and reconstructs missing pixels, while the spectral-wise branch masks random spectral channels and reconstructs missing channels. Our SS-MAE fully exploits the spatial and spectral representations of the input data. Furthermore, to complement local features in the training stage, we add two lightweight CNNs for feature extraction. Both global and local features are taken into account for feature modeling. To demonstrate the effectiveness of the proposed SS-MAE, we conduct extensive experiments on three publicly available datasets. Extensive experiments on three multi-source datasets verify the superiority of our SS-MAE compared with several state-of-the-art baselines. The source codes are available at https://github.com/summitgao/SS-MAE.

show abstract

Section: B Self-supervised Learningmentioning

confidence: 99%