ULMR: An Unsupervised Learning Framework for Mismatch Removal

Abstract

Due to radiometric and geometric distortions between images, mismatches are inevitable. Thus, a mismatch removal process is required for improving matching accuracy. Although deep learning methods have been proved to outperform handcraft methods in specific scenarios, including image identification and point cloud classification, most learning methods are supervised and are susceptible to incorrect labeling, and labeling data is a time-consuming task. This paper takes advantage of deep reinforcement leaning (DRL) and proposes a framework named unsupervised learning for mismatch removal (ULMR). Resorting to DRL, ULMR firstly scores each state–action pair guided by the output of classification network; then, it calculates the policy gradient of the expected reward; finally, through maximizing the expected reward of state–action pairings, the optimal network can be obtained. Compared to supervised learning methods (e.g., NM-Net and LFGC), unsupervised learning methods (e.g., ULCM), and handcraft methods (e.g., RANSAC, GMS), ULMR can obtain higher precision, more remaining correct matches, and fewer remaining false matches in testing experiments. Moreover, ULMR shows greater stability, better accuracy, and higher quality in application experiments, demonstrating reduced sampling times and higher compatibility with other classification networks in ablation experiments, indicating its great potential for further use.

A Survey on the Deep Learning-Based Mismatch Removal: Principles and Methods

“…The optimal DNNs can be obtained by directly maximizing Equation (34). Unsupervised Learning for Mismatch Removal (ULMR) [68] analogizes the mismatch removal problem to playing games and applies reinforcement learning (RL) [154,155] to solve it. From the perspective of RL, the putative matches can be seen as states and the sampling processes are actions.…”

“…That is, the maximal cluster of matches in high dimensional space form an inlier set under the constraint of the correct geometrical model. Thus, to construct an unsupervised learning framework, the first step is applying PINs to project coordinates of matches to high dimensional space (i.e., extracting features of matches); then modulating these features to output matching probabilities (weights); finally, utilizing resampling methods [68] or weighted regression methods [69] to obtain a maximal consensus matching set.…”

MESAC: Learning to Remove Mismatches via Maximizing the Expected Score of Sample Consensuses

“…Following RANSAC's scheme, Neural-guided RANSAC (NG-RANSAC) [48] considers EG constraints of matching points and uses score-function estimator to make RANSAC differentiable. The EG constraints are also adopted by Unsupervised Learning for Mismatch Removal (ULMR) [49] which brows idea from deep reinforcement learning. ULMR regards the mismatch removal problem as a game, and the optimal game player can mine the maximum matching inliers.…”

“…Unsupervised Learning of Consensus Maximization (ULCM) [50] represents transformation of matching inliers by Vandermonde matrix [51], and constructs a loss function only using the knowledge about the polynomial structure of the transformation. Though ULCM avoids directly using score-function estimator to obtain the maximal consensus, it hard to be trained because of the unsmooth surface of the loss function, and experiments show it can only train PINs in matching point sets which have an equally inlier rate [49].…”

“…The above reasons lead to the term 1 µ I(y j ) in Equation ( 7) being a very big number, resulting in a gradient explosion problem. That is why NG-RANSAC [48] and ULMR [49] use a supervised pretrained process to guarantee the convergence of training.…”