Variance Consistency Learning: Enhancing Cross-Modal Knowledge Distillation for Remote Sensing Image Classification

Abstract

Vision Transformers (ViTs) have demonstrated exceptional accuracy in classifying remote sensing images (RSIs). However, existing knowledge distillation (KD) methods for transferring representations from a large ViT to a more compact Convolutional Neural Network (CNN) have proven ineffective. This limitation significantly hampers the remarkable generalization capability of ViTs during deployment due to their substantial size. Contrary to common beliefs, we argue that domain discrepancies along with the RSI inherent natures constrain the effectiveness and efficiency of cross-modal knowledge transfer. Consequently, we propose a novel Variance Consistency Learning (VCL) strategy to enhance the efficiency of the cross-modal KD process, implemented through a plug-and-plug module within a ViTteachingCNN pipeline. We evaluated our student model, termed VCL-Net, on three RSI datasets. The results reveal that VCL-Net exhibits superior accuracy and a more compact size compared to 33 other state-of-the-art methods published in the past three years. Specifically, VCL-Net surpasses other KD-based methods with a maximum improvement in accuracy of 22% across different datasets. Furthermore, the visualization analysis of model activations reveals that VCL-Net has learned long-range dependencies of features from the ViT teacher. Moreover, the ablation experiments suggest that our method has reduced the time costs in the KD process by at least 75%. Therefore, our study offers a more effective and efficient approach for cross-modal knowledge transfer when addressing domain discrepancies.