Adaptive Self-Supervised SAR Image Registration With Modifications of Alignment Transformation

Mao, Shasha; Yang, Jinyuan; Gou, Shuiping; Lü, Kai; Jiao, Licheng; Xiong, Tao; Xiong, Lin

doi:10.1109/tgrs.2023.3246964

Cited by 6 publications

(9 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DINO-MM [37] achieved SAR-optical image representation based on DINO network, which combined heterologous image pairs by connecting all channels to a uniform input, and enhances the data by randomly masking out channels of one modality. AdaSSIR [38] mined the potential features of the keypoints from two images. It treats each keypoint as an independent category, converting keypoints from one image to the other to construct training and test samples, ultimately achieving image alignment.…”

Section: Related Workmentioning

confidence: 99%

Contrastive Learning of Multimodal Consistency Feature Representation for Remote Sensing Image Registration

Han,

Lv,

Wang

et al. 2024

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

Feature representation is a crucial issue in multimodal image registration. The handcrafted features extracted by traditional methods are highly sensitive to nonlinear radiation differences, while supervised learning methods are limited by deficient labeled samples in the remote sensing field. Therefore, this paper proposes a consistency feature representation learning method for multimodal image registration, which involves mapping data into a common feature space to realize the accurate alignment of remote sensing images. First, a contrastive network with a spatial attention mechanism is driven to enhance the capability to highlight high-level features of images. Second, a positive sample augmentation strategy is implemented with contrastive loss, which helps the model learn the inherent features better, and imposes constraints on the sample similarity to optimize the feature projection. Finally, a multimodal image registration framework is proposed to enhance the stability of feature matching. The proposed framework achieves accurate feature extraction and consistency feature description for multimodal images, ensuring robustness against nonlinear radiometric differences. Experimental results demonstrate that the proposed method obtains more reliable registration results on the SEN1-2 dataset. Furthermore, the proposed algorithm achieves superior performance on data from other modalities, indicating strong generalization ability.

show abstract

Section: Related Workmentioning

confidence: 99%

Contrastive Learning of Multimodal Consistency Feature Representation for Remote Sensing Image Registration

Han,

Lv,

Wang

et al. 2024

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

show abstract

“…Fang Shang [24] constructed position vectors and change vectors that cleverly characterize image pixels and classified Polarimetric Synthetic Aperture Radar (PolSAR) images of complex terrain by a Quaternion Neural Network (QNN), which is not influenced by height information. Moreover, advanced techniques integrate self-learning with SIFT feature points for near-subpixel-level registration [7], employ deep forest models to enhance robustness [13], utilize unsupervised learning frameworks for multiscale registration [25][26][27], and leverage Transformer networks for efficient and accurate registration [28][29][30][31][32][33]. Deng, X.…”

Section: Deep Learningmentioning

confidence: 99%

“…This approach effectively circumvents the challenge of constructing matched-point pairs typically encountered in two-classification registration models. In a similar vein, S. Mao [31] introduced an adaptive self-supervised SAR-image-registration method that achieved comparable results. Meanwhile, Li, B.…”

Section: Deep Learningmentioning

confidence: 99%

A Sub-Second Method for SAR Image Registration Based on Hierarchical Episodic Control

Zhou,

Wang,

et al. 2023

Remote Sensing

View full text Add to dashboard Cite

For Synthetic Aperture Radar (SAR) image registration, successive processes following feature extraction are required by both the traditional feature-based method and the deep learning method. Among these processes, the feature matching process—whose time and space complexity are related to the number of feature points extracted from sensed and reference images, as well as the dimension of feature descriptors—proves to be particularly time consuming. Additionally, the successive processes introduce data sharing and memory occupancy issues, requiring an elaborate design to prevent memory leaks. To address these challenges, this paper introduces the OptionEM-based reinforcement learning framework to achieve end-to-end SAR image registration. This framework outputs registered images directly without requiring feature matching and the calculation of the transformation matrix, leading to significant processing time savings. The Transformer architecture is employed to learn image features, while a correlation network is introduced to learn the correlation and transformation matrix between image pairs. Reinforcement learning, as a decision process, can dynamically correct errors, making it more-efficient and -robust compared to supervised learning mechanisms such as deep learning. We present a hierarchical reinforcement learning framework combined with Episodic Memory to mitigate the inherent problem of invalid exploration in generalized reinforcement learning algorithms. This approach effectively combines coarse and fine registration, further enhancing training efficiency. Experiments conducted on three sets of SAR images, acquired by TerraSAR-X and Sentinel-1A, demonstrated that the proposed method’s average runtime is sub-second, achieving subpixel registration accuracy.

show abstract

“…In order to weaken the inherent diversities between two SAR images, a new strategy [25] is applied to transform key points from one image to the other image and then capture all sub-images (corresponding to all key points) from one same image. Here, a simple method is applied to obtain the initial transformation matrix as a rough registration.…”

Section: Constructing Samples-based Key Pointsmentioning

confidence: 99%

“…This essential characteristic leads us to a worthwhile thought: why did we not directly consider multiple key points as multiple classes to construct a multi-classification deep model for SAR image registration, instead of a two-classification deep model? Recently, Mao et al [25] proposed an adaptive self-supervised SAR image registration method, which utilized each key point as an independent instance to train the self-supervised deep model and then compared the latent features of each key point with other points to search for the final matched-point pairs. Therefore, inspired by these, we aim to design the SAR image registration method from the perspective of multiple classification (discriminating multiple key points), where each key point is considered as an independent class, abandoning the idea of constructing a two-classification model (discriminating matched and non-matched pairs).…”

Section: Introductionmentioning

confidence: 99%

Multi-Class Double-Transformation Network for SAR Image Registration

et al. 2023

Self Cite

View full text Add to dashboard Cite

In SAR image registration, most existing methods consider the image registration as a two-classification problem to construct the pair training samples for training the deep model. However, it is difficult to obtain a mass of given matched-points directly from SAR images as the training samples. Based on this, we propose a multi-class double-transformation network for SAR image registration based on Swin-Transformer. Different from existing methods, the proposed method directly considers each key point as an independent category to construct the multi-classification model for SAR image registration. Then, based on the key points from the reference and sensed images, respectively, a double-transformation network with two branches is designed to search for matched-point pairs. In particular, to weaken the inherent diversity between two SAR images, key points from one image are transformed to the other image, and the transformed image is used as the basic image to capture sub-images corresponding to all key points as the training and testing samples. Moreover, a precise-matching module is designed to increase the reliability of the obtained matched-points by eliminating the inconsistent matched-point pairs given by two branches. Finally, a series of experiments illustrate that the proposed method can achieve higher registration performance compared to existing methods.

show abstract

Adaptive Self-Supervised SAR Image Registration With Modifications of Alignment Transformation

Cited by 6 publications

References 44 publications

Contrastive Learning of Multimodal Consistency Feature Representation for Remote Sensing Image Registration

Contrastive Learning of Multimodal Consistency Feature Representation for Remote Sensing Image Registration

A Sub-Second Method for SAR Image Registration Based on Hierarchical Episodic Control

Multi-Class Double-Transformation Network for SAR Image Registration

Contact Info

Product

Resources

About