Variational method for joint optical flow estimation and edge-aware image restoration

Abstract

The most popular optical flow algorithms rely on optimizing the energy function that integrates a data term and a smoothness term. In contrast to this traditional framework, we derive a new objective function that couples optical flow estimation and image restoration. Our method is inspired by the recent successes of edge-aware constraints (EAC) in preserving edges in general gradient domain image filtering. By incorporating an EAC image fidelity term (IFT) in the conventional variational model, the new energy function can simultaneously estimate optical flow and restore images with preserved edges, in a bidirectional manner. For the energy minimization, we rewrite the EAC into gradient form and optimize the IFT with Euler-Lagrange Equations. We can thus apply the image restoration by analytically solving a system of linear equations. Our EAC-combined IFT is easy to implement and can be seamlessly integrated into various optical flow functions suggested in literature. Extensive experiments on public optical flow benchmarks demonstrate that our method outperforms the current state-of-the-art in optical flow estimation and image restoration.Keywords: optical flow, image sequence restoration, edge preserving, efficient numerical solver. 15 varies smoothly over the flow field. In practice, however, these two constraints are often violated, which prevents the variational method to robustly treat image noise, handle illumination changes and large displacements, and to preserve flow discontinuities.Various extensions and improvements have been proposed to handle these 20 problems [10]. In general, these variations can be classified into three categories [12]: pre-processing of the input images, modification of the variational formulation and post-processing of the flow field.Pre-processing is often applied to remove image artifacts of the input images, including noise filtering [13, 14] and deblurring [15]. 25Modifications are the second category of variations of the traditional formulation. They are aimed at improving the data term to make the algorithm more 2 resistant to noise [16,17], more robust under illumination changes [18,19, 20], and more capable to deal with large displacements [10, 1,21]. In addition, refining the smoothness term to preserve motion discontinuity [22]. 30 One challenge for optical flow computation is to deal with large displacements. Recently, there has been an interest in using feature matching to assist the variational methods to address this problem. By matching image features, reliable correspondence information between two images can be obtained [21].The feature matching and variational methods are complementary, but com-35 bining them is not straightforward. Feature matching exploits a discrete space of admissible correspondences, whereas the variational model imposes linearization during the differential optimization procedure [23]. Brox et al. [10] adds a feature matching constraint into the variational framework. In this way, correspondences from descriptor matching are obtained, wh...

Adaptive dual fractional‐order variational optical flow model for motion estimation

“…The well‐known BCA in the data term is not sufficient for coping with real scenes which include complex illumination changes and other challenging situations. Several other assumptions have been introduced into the data term, such as the gradient [17], the Laplacian [18] and the Hessian [19] constancy assumptions. However, integer order derivative‐based methods depend too heavily on the difference of pixel values, leading to inaccurate optical flow values when large texture‐less regions are encountered.…”

A Robust Functional Minimization Technique to Protect Image Details from Disturbances

“…Moreover, the solution of the models can ensure image smoothness and details preservation abilities, those abilities are also reported as in the PDE-based methods [20,22]. Similarly, energy minimization or variationalbased approaches such as [23][24][25][27][28][29] using in image enhancement problems. The nonconvexbased smoothing approaches such as [30][31][32][33] can also preserve image details.…”

An Improved Fractional-Order Variational Optical Flow Model Combining Structure Tensor

“…Several other constraints on the data term were proposed to improve the performance of the variational optical flow model, such as gradient constancy constraint [2], Laplacian constancy constraint [3], and Hessian constancy constraint [4], but these constraints depend heavily on the intensity difference. Illumination invariant descriptors such as binarybased [5], real value-based [6,7], and neighborhood-based [8,9] were applied to substitute the brightness constancy constraint.…”