Learning a Confidence Measure for Optical Flow

Abstract

We present a supervised learning-based method to estimate a per-pixel confidence for optical flow vectors. Regions of low texture and pixels close to occlusion boundaries are known to be difficult for optical flow algorithms. Using a spatiotemporal feature vector, we estimate if a flow algorithm is likely to fail in a given region. Our method is not restricted to any specific class of flow algorithm and does not make any scene specific assumptions. By automatically learning this confidence, we can combine the output of several computed flow fields from different algorithms to select the best performing algorithm per pixel. Our optical flow confidence measure allows one to achieve better overall results by discarding the most troublesome pixels. We illustrate the effectiveness of our method on four different optical flow algorithms over a variety of real and synthetic sequences. For algorithm selection, we achieve the top overall results on a large test set, and at times even surpass the results of the best algorithm among the candidates.

Ensemble Classifier for Combining Stereo Matching Algorithms

“…Compared to the work of Mac Aodha et al [26] on optical flow, we claim that our formulation using the oneagainst-all classifiers is more effective in capturing the agreement and disagreement between different matchers by not assuming that they are independent. Evidence for this can be seen by comparing ensembles M and N in Table 2.…”

“…As shown in Section 5, we do not consider the matchers independent but encode whether they agree or not. This provides valuable information that makes our final disparity assignments consistently better than all input matchers, while the combination of [26] is second best on all sequences. In subsequent work, Mac Aodha and Brostow [25] proposed cost sensitive learning for selecting among multiple experts.…”

“…The work of Mac Aodha et al [26] is similar to ours, as it uses a multi-class classifier that selects among four state of the art methods for optical flow estimation. They also make decisions at the pixel level using a random forest.…”

Lessons and Insights from Creating a Synthetic Optical Flow Benchmark

“…Compared to the complex scenes [5]; Complex sequence from [6]; Grove3 sequence from [1]; 009 Crates1 sequence from [7]; Bamboo3 sequence from MPI-Sintel [2].…”

“…UCL-Flow [7] introduces 20 new image pairs with strong rigid object motion and more complex light situations with colored light sources and ambient lighting. Its intended use is to train a mechanism to locally select the best optical flow algorithm, given a set of image features.…”