Determining Occlusions from Space and Time Image Reconstructions

Abstract

The problem of localizing occlusions between consecutive frames of a video is important but rarely tackled on its own. In most works, it is tightly interleaved with the computation of accurate optical flows, which leads to a delicate chicken-and-egg problem. With this in mind, we propose a novel approach to occlusion detection where visibility or not of a point in next frame is formulated in terms of visual reconstruction. The key issue is now to determine how well a pixel in the first image can be "reconstructed" from co-located colors in the next image. We first exploit this reasoning at the pixel level with a new detection criterion. Contrary to the ubiquitous displaced-framedifference and forward-backward flow vector matching, the proposed alternative does not critically depend on a precomputed, dense displacement field, while being shown to be more effective. We then leverage this local modeling within an energy-minimization framework that delivers occlusion maps. An easy-to-obtain collection of parametric motion models is exploited within the energy to provide the required level of motion information. Our approach outperforms state-of-the-art detection methods on the challenging MPI Sintel dataset.

Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimation

“…Humayun et al [16] determine occlusions post-hoc by training a classifier on a broad spectrum of visual features and precomputed optical flow. Pérez-Rúa et al [32] do not require a dense optical flow field, but motion candidates, which are used to determine if a "plausible reconstruction" exists. Many other methods try to estimate optical flow and occlusions jointly.…”

“…Since Sintel [7] does not provide the ground-truth backward flow, we additionally report numbers on FlyingThings3D [29]. The results show that contrary to literature [32,27,16], occlusion estimation is even possible from just the two images. Providing the optical flow, too, clearly improves the results (by a factor of 1 20 ) yields noise for small displacements during optimization.…”

Phase-Based Nonrigid Deformation for Digital Subtraction Angiography

“…One approach to jointly solve the image registration and change detection is optical flow estimation with occlusion handling [3]. An up-to-date method [21] presents a novel occlusion detection criterion, which does not critically depend on a precomputed dense motion flow field. Hur and Roth [20] recently proposed to utilize symmetry properties of the flow and occlusion; i.e., the forward-backward consistency and occlusion-disocclusion symmetry in the energy function.…”

Hierarchical motion decomposition for dynamic scene parsing

“…It is important to note here that the motion models describe perregion motion characteristics, in contrast to the instrumental motion field f , which is an unstructured dense reference. Reasoning on piecewise parametric motion models have been successfully used recently to solve intricate problems like optical flow [15] and occlusion detection [16].…”