Optical Flow with Semantic Segmentation and Localized Layers

Abstract

Existing optical flow methods make generic, spatially homogeneous, assumptions about the spatial structure of the flow. In reality, optical flow varies across an image depending on object class. Simply put, different objects move differently. Here we exploit recent advances in static semantic scene segmentation to segment the image into objects of different types. We define different models of image motion in these regions depending on the type of object. For example, we model the motion on roads with homographies, vegetation with spatially smooth flow, and independently moving objects like cars and planes with affine motion plus deviations. We then pose the flow estimation problem using a novel formulation of localized layers, which addresses limitations of traditional layered models for dealing with complex scene motion. Our semantic flow method achieves the lowest error of any published monocular method in the KITTI-2015 flow benchmark and produces qualitatively better flow and segmentation than recent top methods on a wide range of natural videos.

Volumetric Flow Estimation for Incompressible Fluids Using the Stationary Stokes Equations

“…Recently, 2D optical flow benchmarks have been dominated by label-based methods [7,24], propagation methods [4,18], neural regression networks [10] and models that exploit scene-specific properties like semantics [35,3]. Most of these models do not scale well to the volumetric domain and struggle heavily with memory consumption.…”

Temporally Consistent Motion Segmentation From RGB‐D Video

“…They point out the advantage of obtaining consistent segmentations over entire video sequences, which is also a goal in our approach. Recent techniques jointly solve for RGB video segmentation and optical flow [SLSJB16, TYB16]. Learning‐based approaches have also become popular for motion segmentation [FAFM15] and optical flow [DFI*15], and Sevilla‐Lara et al .…”

“…Learning‐based approaches have also become popular for motion segmentation [FAFM15] and optical flow [DFI*15], and Sevilla‐Lara et al . [SLSJB16] leverage semantic segmentation using deep convolutional neural networks (CNNs) in their approach. Similar to these techniques, we also obtain both a segmentation and motion estimates.…”

Semantic filtering for video stabilization

“…It is worth noting that this transition causes inaccurate mo- tion vector estimation at both sides of the motion boundary, since the two motion fields influence each other. However, recent work on optical flow estimation has leveraged the use of additional information to improve flow precision, particularly at object boundaries [18,19].…”