Continual Occlusion and Optical Flow Estimation

Abstract

Two optical flow estimation problems are addressed: i) occlusion estimation and handling, and ii) estimation from image sequences longer than two frames. The proposed ContinualFlow method estimates occlusions before flow, avoiding the use of flow corrupted by occlusions for their estimation. We show that providing occlusion masks as an additional input to flow estimation improves the standard performance metric by more than 25% on both KITTI and Sintel. As a second contribution, a novel method for incorporating information from past frames into flow estimation is introduced. The previous frame flow serves as an input to occlusion estimation and as a prior in occluded regions, i.e. those without visual correspondences. By continually using the previous frame flow, ContinualFlow performance improves further by 18% on KITTI and 7% on Sintel, achieving top performance on KITTI and Sintel.

Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation

“…We test the accuracy of our IRR-PWC on the public Sintel [10] and KITTI [17,41] benchmarks. When fine-tuning, we use the robust training loss as in [24,52,53] [53] (1.45) (7.59%) 7.72% LiteFlowNet [24] (1.62) (5.58%) 9.38% PWC-Net [52] (2.16) (9.80%) 9.60% ContinualFlow ROB † § [42] --10.03% MirrorFlow [25] -9.98% 10.29% FlowNet2 [26] (2.30) (8.61%) 10.41% vations are that our model (i) converges much faster than the baseline and (ii) overfits to the training split less, demonstrating much better accuracy on the test set despite slightly higher error on training split. This highlights the benefit of our IRR scheme: better generalization even on the training domain as well as across datasets.…”

Traditional and modern strategies for optical flow: an investigation

“…Based on machine learning principles, these algorithms learn to compute optical flow from a pair of input images. In recent years convolutional networks have been used to estimate the optical flow with promising results [23][24][25][26][27][28]30]. Convolutional neural nets can go through supervised or unsupervised training for per-pixel image classification.…”

“…Among modern researchers [67] proposed a variational model with a self-adaptive weight energy function and non-local term. In this sense the most powerful schemes are those utilizing occlusions as supplementary evidence to compute optical flow such as MirrorFlow [22], ContinualFlow [28], SelFlow [30]. This approach is contrary to the classical methods applying forward/backward inconsistency check and discarding occlusions as outliers.…”

“…This was achieved through borrowing many popular ideas from traditional methods and implanting into deep learning schemes, stimulating a great amount of work. SpyNet [25] combines coarse-to-fine approach of traditional methods and deep learning methods, LiteFlow [26] uses traditional brightness inconsistency map to tackle occlusions, PWC-Net [27] integrates traditional stereo matching, feature extraction and cost volume with deep learning, Continual Flow [28] combines occlusions and cost volume together with optical flow, MFF [29] fuses the warped optical flow from previous frame with the current optical flow estimate, SelFlow [30] uses flow estimation from non-occluded pixels for self-learning, IRR [31] learns to refine its previous estimate by iteratively re-using the same network block with shared weights. At present, optical flow schemes based on deep learning have completely outperformed the traditional methods in accuracy and run time.…”

Improved Discrete Optical Flow Estimation With Triple Image Matching Cost

“…They construct past cost volume and future cost volume with three frames and leverage convolutional neural network to reason occlusion. Neoral et al [42] also estimate occlusion masks by introducing the previous frame flow and named it Con-tinualFlow. Ren et al [43] use a neural network to fuse optical flows of different moments depending on longer-term temporal cues.…”