Motion vector refinement for FRUC using saliency and segmentation

Abstract

Motion-Compensated Frame Interpolation (MCFI) is a technique used extensively for increasing the temporal frequency of a video sequence. In order to obtain a high quality interpolation, the motion field between frames must be well-estimated. However, many current techniques for determining the motion are prone to errors in occlusion regions, as well as regions with repetitive structure. An algorithm is proposed for improving both the objective and subjective quality of MCFI by refining the motion vector field. A Discriminant Saliency classifier is employed to determine regions of the motion field which are most important to a human observer. These regions are refined using a multi-stage motion vector refinement which promotes candidates based on their likelihood given a local neighborhood. For regions which fall below the saliency threshold, frame segmentation is used to locate regions of homogeneous color and texture via Normalized Cuts. Motion vectors are promoted such that each homogeneous region has a consistent motion. Experimental results demonstrate an improvement over previous methods in both objective and subjective picture quality.

Statistical search range adaptation solution for effective frame rate up‐conversion

“…Since human observers typically focus their visual attention on small regions of the video frame that appear interesting, Jacobson et al [16,17] proposed a discriminant saliency classifier-based MVF refinement solution where the salience regions were firstly classified and considered as important regions. The MVs in these regions were refined using a multi-stage MV refinement [12].…”

A Spatiotemporal Saliency Model for Video Surveillance

“…Instead, the coherent perception of moving objects, even when the vertex motions are incoherent and the background motion cannot be easily explained by a physical geometric transformation, suggests that both motion saliency and perceptual organization are driven by measurements of local motion contrast [20,41]. To account for the variability between the state at time t and the sequence of past states and to make the spatiotemporal features robust enough to handle complex dynamic backgrounds we propose to analyze the temporal consistency of motion vectors.…”