Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses

Abstract

We present Neural-Guided RANSAC (NG-RANSAC), an extension to the classic RANSAC algorithm from robust optimization. NG-RANSAC uses prior information to improve model hypothesis search, increasing the chance of finding outlier-free minimal sets. Previous works use heuristic side information like hand-crafted descriptor distance to guide hypothesis search. In contrast, we learn hypothesis search in a principled fashion that lets us optimize an arbitrary task loss during training, leading to large improvements on classic computer vision tasks. We present two further extensions to NG-RANSAC. Firstly, using the inlier count itself as training signal allows us to train neural guidance in a self-supervised fashion. Secondly, we combine neural guidance with differentiable RANSAC to build neural networks which focus on certain parts of the input data and make the output predictions as good as possible. We evaluate NG-RANSAC on a wide array of computer vision tasks, namely estimation of epipolar geometry, horizon line estimation and camera re-localization. We achieve superior or competitive results compared to state-of-the-art robust estimators, including very recent, learned ones.

Let's Take This Online: Adapting Scene Coordinate Regression Network Predictions for Online RGB-D Camera Relocalisation

“…As per [7], we also compare to a number of other well-known methods that are capable of making use of the 3D model [5,35,59]. Note that, in common with other learning-based methods [5,7,8], we ignore the Street scene, for which our method too was unable to produce reasonable results (the SfM reconstruction in the dataset appears to be of poor quality for this scene [8]).…”

“…(iv) Local regression methods generally use regression forests [63,27,69,6,48,13,49,50,12], neural networks [5,7,18,42,43,8], or a mix of the two [46] to predict the scene coordinates of pixels in the input image. They then pass these correspondences to PnP/Kabsch and RANSAC.…”

Hierarchical Scene Coordinate Classification and Regression for Visual Localization

“…Instead of learning the en-tire pipeline, scene coordinate regression methods learn the first stage of the pipeline in the structure-based approaches. Namely, either a random forest [4,12,13,20,30,32,33,50,57] or a neural network [3,5,6,7,9,10,11,27,28,30] is trained to directly predict 3D scene coordinates for the pixels and thus the 2D-3D correspondences are established. These methods do not explicitly rely on feature detection, description and matching, and are able to provide correspondences densely.…”

Neural3D: Light-weight Neural Portrait Scanning via Context-aware Correspondence Learning

“…DSAC [9] adopts a probabilistic selecting process to make it differential so that the complete process can be trained in an end-to-end manner, while Marginalizing Sample Consensus (MAGSAC) [6] proposes to find the optimal model through weighted least-squares fitting without estimation of an inlier-outlier threshold. Recently, Neural-Guided RANSAC(NG-RANSAC) [10] employs deep networks to first estimate the confidence of the putative correspondences being inliers to guide the matching process with improved model hypothesis searching. As the most closely related to our approach, PointCN [27] employs the PointNet-like [33,34] architecture to classify every pair of correspondences as either inlier or outlier and then uses the weighted eight-point algorithm for essential matrix estimation.…”

“…We train our network on the subsets of the two datasets and evaluate the correspondence estimation performance on the other subsets of the same scenes as the test set and the datasets from the other scenes as the test set. For thorough evaluation, we compare against both the traditional baseline technique and the state-of-the-art approaches including [27], [54] and − [10]. We adopt the standard angular difference between the estimation and the ground truth and measure the mean average precision (mAP) under accuracy a threshold (5 • ) for both rotation and translation.…”