Learning to Detect Ground Control Points for Improving the Accuracy of Stereo Matching

Abstract

While machine learning has been instrumental to the ongoing progress in most areas of computer vision, it has not been applied to the problem of stereo matching with similar frequency or success. We present a supervised learning approach for predicting the correctness of stereo matches based on a random forest and a set of features that capture various forms of information about each pixel.We show highly competitive results in predicting the correctness of matches and in confidence estimation, which allows us to rank pixels according to the reliability of their assigned disparities. Moreover, we show how these confidence values can be used to improve the accuracy of disparity maps by integrating them with an MRF-based stereo algorithm. This is an important distinction from current literature that has mainly focused on sparsification by removing potentially erroneous disparities to generate quasi-dense disparity maps.

Ensemble Classifier for Combining Stereo Matching Algorithms

“…A set of active matchers is selected from a potentially very large initial pool and random forest classifiers are trained to select the disparity that is most likely to be correct for each pixel of the left image. The classifiers, even with a small number of features compared to [14,34], are always able to surpass the accuracy of the best matcher in the active set. Our method combines eight matchers and achieves an error rate before post-processing (5.82%) that is significantly smaller than that of the best individual matcher (8.06%).…”

“…It uses a number of features computed on the images, disparity maps and matching cost volume and a random forest classifier that is able to reject false matches with high accuracy. Spyropoulos et al [34] use a random forest to predict match correctness and to select ground control points which provide constraints that improve the overall accuracy. Zbontar and LeCun [40] trained a convolutional neural network (CNN) to predict whether two image patches match or not.…”

Confidence Estimation for Superpixel-Based Stereo Matching

“…Haeusler et al [14] trained a random forest classifier using a number of confidence measures as features to make predictions about the correctness of the outputs of the semi-global matching algorithm. Spyropoulos et al [34] used a similar classification approach, but also demonstrated that such a classifier can be used to select ground control points, which in turn can help improve the accuracy of the input disparity maps. Park and Yoon [29] also use a number of confidence measures as features in a random forest classifier that predicts the correctness of WTA disparities.…”

“…Recent research on confidence estimation [14,34,29] integrates multiple sources of information within a discriminative learning framework, instead of relying on a single feature for each pixel [11,16]. As expected, taking into account multiple features leads to improved performance since different failure modes can be detected, while individual features typically respond to one failure mode.…”

Patch Based Confidence Prediction for Dense Disparity Map

“…The patch consists in a two channels. 1st channel is coming from an idea that neighboring pixels on a disparity map D 1 which have consistent disparities are more likely to be correct matching [7]. In 2nd channel, a disparity D 2 from another image is considered such that the matches from left to right image should be consistent with those from right to left [1].…”