SemFlow: Semantic-Driven Interpolation for Large Displacement Optical Flow

Wang, Xianshun; Zhu, Dongchen; Liu, Yanqing; Ye, Xiaoqing; Li, Jiamao; Zhang, Xiaolin

doi:10.1109/access.2019.2911021

Cited by 8 publications

(6 citation statements)

References 31 publications

(61 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cheng et al [ 11 ] tried to capture the interdependence between the tasks of semantic segmentation and optical flow estimation, jointly training a network for video object segmentation and optical flow, while employing late feature fusion. Wang et al [ 40 ] used semantic masks for superpixel refinement and built a semantic-guided superpixel distance metric in order to improve sparse matching and flow estimation accuracy at object boundaries.…”

Section: Related Workmentioning

confidence: 99%

Implicit and Explicit Regularization for Optical Flow Estimation

Karageorgos

Dimou

Álvarez

et al. 2020

Sensors

View full text Add to dashboard Cite

In this paper, two novel and practical regularizing methods are proposed to improve existing neural network architectures for monocular optical flow estimation. The proposed methods aim to alleviate deficiencies of current methods, such as flow leakage across objects and motion consistency within rigid objects, by exploiting contextual information. More specifically, the first regularization method utilizes semantic information during the training process to explicitly regularize the produced optical flow field. The novelty of this method lies in the use of semantic segmentation masks to teach the network to implicitly identify the semantic edges of an object and better reason on the local motion flow. A novel loss function is introduced that takes into account the objects’ boundaries as derived from the semantic segmentation mask to selectively penalize motion inconsistency within an object. The method is architecture agnostic and can be integrated into any neural network without modifying or adding complexity at inference. The second regularization method adds spatial awareness to the input data of the network in order to improve training stability and efficiency. The coordinates of each pixel are used as an additional feature, breaking the invariance properties of the neural network architecture. The additional features are shown to implicitly regularize the optical flow estimation enforcing a consistent flow, while improving both the performance and the convergence time. Finally, the combination of both regularization methods further improves the performance of existing cutting edge architectures in a complementary way, both quantitatively and qualitatively, on popular flow estimation benchmark datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

Implicit and Explicit Regularization for Optical Flow Estimation

Karageorgos

Dimou

Álvarez

et al. 2020

Sensors

View full text Add to dashboard Cite

show abstract

“…The quality of frame interpolation depends heavily on the accuracy of optical flow. In recent years, despite the great progresses in optical flow estimation [14], [15], [27]- [33], there are still some difficulties, such as obvious occlusion, motion boundary, large motion and motion blur. Mahajan et al [34] compute paths in the input frames and copy pixel gradients along them to the interpolated frame, and then synthesize the intermediate frame via Poisson reconstruction.…”

Section: Related Workmentioning

confidence: 99%

“…Dosovitskiy et al [28] develop two network architectures: FlowNetS and FlowNetC, which show the feasibility of estimating optical flow from raw frames using a U-Net [41] architecture. Wang et al [33] presents a semanticguided interpolation scheme (SemFlow) to handle motion boundaries and occlusions in large displacement optical flow. Recently, the combination of classical principles of optical flow with the network architecture [14], [15] achieves better results and requires less computation.…”

Section: Related Workmentioning

confidence: 99%

Multi-Frame Pyramid Refinement Network for Video Frame Interpolation

2019

View full text Add to dashboard Cite

Video frame interpolation aims at synthesizing new video frames in-between existing frames to generate higher frame rate video. Current methods usually use two adjacent frames to generate intermediate frames, but sometimes fail to handle challenges like large motion, occlusion, and motion blur. This paper proposes a multi-frame pyramid refinement network to effectively use spatio-temporal information contained in multiple frames (more than two). There are three technical contributions in the proposed network. First, a special coarse-to-fine framework is proposed to refine optical flows in-between multiple frames with residual flows at each pyramid level. Therefore, large motion and occlusion can be effectively estimated. Second, a 3D U-net feature extractor is used to excavate spatio-temporal context and restore texture, which tend to disappear at course pyramid levels. Third, a multi-step perceptual loss is adopted to preserve more details in intermediate frame. It is worth mentioning that our approach can be easily extended to multi-frame interpolation. Our network is trained end-to-end using more than 80K collected frame groups (25 frames per group). Experimental results on several independent datasets show that our approach can effectively handle challenging cases, and perform consistently better than other state-of-the-art methods.INDEX TERMS Video frame interpolation, multiple frames, spatio-temporal information, optical flow, coarse-to-fine framework, deep learning.

show abstract

“…The problems of interpolation and extrapolation have a long history in mathematics and computer science. In high-level computer vision, interpolation finds its application in various problems like motion estimation in 2D (optical flow) [1,2,11,14,15,20,30,31,33,43,50], 3D (scene flow) [36,37], or depth completion [6,17,26,40,41]. These methods in turn are applied in robot navigation, advanced driver assistance systems (ADAS), surveillance, and many others.…”

Section: Introductionmentioning

confidence: 99%

SSGP: Sparse Spatial Guided Propagation for Robust and Generic Interpolation

Schuster¹,

Wasenmüller²,

Unger³

et al. 2020

Preprint

View full text Add to dashboard Cite

Interpolation of sparse pixel information towards a dense target resolution finds its application across multiple disciplines in computer vision. State-of-the-art interpolation of motion fields applies model-based interpolation that makes use of edge information extracted from the target image. For depth completion, data-driven learning approaches are widespread. Our work is inspired by latest trends in depth completion that tackle the problem of dense guidance for sparse information. We extend these ideas and create a generic cross-domain architecture that can be applied for a multitude of interpolation problems like optical flow, scene flow, or depth completion. In our experiments, we show that our proposed concept of Sparse Spatial Guided Propagation (SSGP) achieves improvements to robustness, accuracy, or speed compared to specialized algorithms.

show abstract

SemFlow: Semantic-Driven Interpolation for Large Displacement Optical Flow

Cited by 8 publications

References 31 publications

Implicit and Explicit Regularization for Optical Flow Estimation

Implicit and Explicit Regularization for Optical Flow Estimation

Multi-Frame Pyramid Refinement Network for Video Frame Interpolation

SSGP: Sparse Spatial Guided Propagation for Robust and Generic Interpolation

Contact Info

Product

Resources

About