Optical and SAR Image Matching Using Pixelwise Deep Dense Features

Zhang, Han; Lei, Lin; Ni, Weijian; Tang, Tao; Jian, Wu; Xiang, Deliang; Kuang, Gangyao

doi:10.1109/lgrs.2020.3039473

Cited by 26 publications

(20 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To qualitatively evaluate the structural features of the pseudo-optical images generated by our network, inspired by the research in [61], a fast Fourier transform (FFT)-accelerated sum of squared differences (SSD) method is used to measure the similarity between the structural features of pseudo-optical images obtained via different methods and those of real optical images. The value of the SSD score plot indicates the offset between image pairs, and a smaller value indicates a higher similarity of their features [62].…”

Section: Discussionmentioning

confidence: 99%

A Dual-Generator Translation Network Fusing Texture and Structure Features for SAR and Optical Image Matching

Nie

Tang

et al. 2022

Remote Sensing

View full text Add to dashboard Cite

The matching problem for heterologous remote sensing images can be simplified to the matching problem for pseudo homologous remote sensing images via image translation to improve the matching performance. Among such applications, the translation of synthetic aperture radar (SAR) and optical images is the current focus of research. However, the existing methods for SAR-to-optical translation have two main drawbacks. First, single generators usually sacrifice either structure or texture features to balance the model performance and complexity, which often results in textural or structural distortion; second, due to large nonlinear radiation distortions (NRDs) in SAR images, there are still visual differences between the pseudo-optical images generated by current generative adversarial networks (GANs) and real optical images. Therefore, we propose a dual-generator translation network for fusing structure and texture features. On the one hand, the proposed network has dual generators, a texture generator, and a structure generator, with good cross-coupling to obtain high-accuracy structure and texture features; on the other hand, frequency-domain and spatial-domain loss functions are introduced to reduce the differences between pseudo-optical images and real optical images. Extensive quantitative and qualitative experiments show that our method achieves state-of-the-art performance on publicly available optical and SAR datasets. Our method improves the peak signal-to-noise ratio (PSNR) by 21.0%, the chromatic feature similarity (FSIMc) by 6.9%, and the structural similarity (SSIM) by 161.7% in terms of the average metric values on all test images compared with the next best results. In addition, we present a before-and-after translation comparison experiment to show that our method improves the average keypoint repeatability by approximately 111.7% and the matching accuracy by approximately 5.25%.

show abstract

Section: Discussionmentioning

confidence: 99%

A Dual-Generator Translation Network Fusing Texture and Structure Features for SAR and Optical Image Matching

Nie

Tang

et al. 2022

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…[33]. Then, they used the Siamese convolutional network to learn pixelwise deep dense features to further improve the robustness of the network [34]. Dou et al designed a generic matching network (GMN) based on the generic adversarial network (GaN) to increase the amount of training data and improve the matching performance [35].…”

Section: Et Al Proposed the Uniform Nonlinear Diffusion-basedmentioning

confidence: 99%

A Robust 3D Density Descriptor Based on Histogram of Oriented Primary Edge Structure for SAR and Optical Image Co-Registration

Ren³

et al. 2022

Remote Sensing

View full text Add to dashboard Cite

The co-registration between SAR and optical images is a challenging task because of the speckle noise of SAR and the nonlinear radiation distortions (NRD), particularly in the one-look situation. In this paper, we propose a novel density descriptor based on the histogram of oriented primary edge structure (HOPES) for the co-registration of SAR and optical images, aiming to describe the shape structure of patches more firm. In order to extract the primary edge structure, we develop the novel multi-scale sigmoid Gabor (MSG) detector and a primary edge fusion algorithm. Based on the HOPES, we propose the co-registration method. To obtain stable and uniform keypoints, the non-maximum suppressed SAR-Harris (NMS-SAR-Harris) and deviding grids methods are used. NMS-SSD fast template matching and fast sample consensus (FSC) algorithm are used to further complete and optimize matching. We use two one-look simulated SAR images to demonstrate that the signal-to-noise ratio (SNR) of MSG is more than 10 dB higher than other state-of-the-stage detectors; the binary edge maps and F-score show that MSG has more accurate positioning performance. Compared with the other state-of-the-stage co-registration methods, the image co-registration results obtained on seven pairs of test images show that, the correct match rate (CMR) and the root mean squared error (RMSE) improve by more than 25% and 15% on average, respectively. It is experimentally demonstrated that the HOPES is robust against speckle noise and NRD, which can effectively improve the matching success rate and accuracy.

show abstract

“…However for optical-SAR image pairs, especially the high resolution ones, a high ratio of mismatches is always obtained, caused by the vast modal disparity in-between. Therefore in recent years, numerous researches have been conducted in the field of optical and SAR sparse feature point matching problem so as to increase the correctly matching rate, including the traditional handcrafted approaches [11][12][13][14][15][16][17][18][19], as well as the learning based methods [20][21][22][23][24][25][26][27][28][29][30][31][32][33][34].…”

Section: Introductionmentioning

confidence: 99%

“…The traditional optical flow algorithms always require brightness constancy assumption, which apparently does not hold for high resolution optical and SAR image pairs. Fortunately, the deep learning technique has shown great potential to learn homogenous features from heterogeneous image pairs [5,[20][21][22][23][24][25][26][27][28][29][30][31][32][33][34]. Furthermore, the deep learning based optical flow methods have recently outperformed the best elaborately designed traditional methods, and also been significantly faster at inference time.…”

Section: Introductionmentioning

confidence: 99%

Optical and SAR Image Dense Registration Using a Robust Deep Optical Flow Framework

Zhang

Lei

et al. 2023

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

The co-registration of optical and SAR imageries is the bottleneck in exploring the complementary information from the two multi-modal datasets. The difficulties lie in not only the complex radiometric relationship between them, but also the distinct geometrical models of the optical and SAR imaging systems, which cause it nontrivial to explicitly depict the spatial relationship between the corresponding image regions when elevation fluctuations exist. This article aims to investigate the optical flow technique for the pixelwise dense registration of the high resolution optical and SAR images, so as to get rid of the outlier removal and geometric mapping procedures, which have to be conducted in the classical image registration approaches that are based on sparse feature point matching. Herein, a deep optical flow framework is designed. Firstly, a dilated feature concatenation method is proposed to enhance the discriminability of the pixelwise features for similarity measurement. An effective network training strategy is used, based on a smoothed flow loss, and also a training dataset that contains simulated elevation fluctuations. Secondly, we propose a self-supervised optical flow finetuning strategy. It incorporates the strength of the blockwise matching approach, which produces better matching precision, into the proposed pixelwise matching procedure. In this way, the accuracy of the optical-SAR dense registration is substantially improved. Extensive experiments conducted on the 1m resolution optical-SAR image pairs of different land-cover types and distinct topographic conditions indicate that, the proposed OSFlowNet-Ft framework is quite robust, and has the potential to perform the optical-SAR image dense registration in practical applications. The Python code of the proposed deep optical flow network will be made available.

show abstract

Optical and SAR Image Matching Using Pixelwise Deep Dense Features

Cited by 26 publications

References 13 publications

A Dual-Generator Translation Network Fusing Texture and Structure Features for SAR and Optical Image Matching

A Dual-Generator Translation Network Fusing Texture and Structure Features for SAR and Optical Image Matching

A Robust 3D Density Descriptor Based on Histogram of Oriented Primary Edge Structure for SAR and Optical Image Co-Registration

Optical and SAR Image Dense Registration Using a Robust Deep Optical Flow Framework

Contact Info

Product

Resources

About