We propose the use of nonlocal operators to define new types of flows and functionals for image processing and elsewhere. A main advantage over classical PDE-based algorithms is the ability to handle better textures and repetitive structures. This topic can be viewed as an extension of spectral graph theory and the diffusion geometry framework to functional analysis and PDE-like evolutions. Some possible application and numerical examples are given, as is a general framework for approximating Hamilton-Jacobi equations on arbitrary grids in high demensions, e.g., for control theory.

This paper explores various aspects of the image decomposition problem using modern variational techniques. We aim at splitting an original image f into two components u and v, where u holds the geometrical information and v holds the textural information. The focus of this paper is to study different energy terms and functional spaces that suit various types of textures. Our modeling uses the total-variation energy for extracting the structural part and one of four of the following norms for the textural part: L 2 , G, L 1 and a new tunable norm, suggested here for the first time, based on Gabor functions. Apart from the broad perspective and our suggestions when each model should be used, the paper contains three specific novelties: first we show that the correlation graph between u and v may serve as an efficient tool to select the splitting parameter, second we propose a new fast algorithm to solve the TV − L 1 minimization problem, and third we introduce the theory and design tools for the TV-Gabor model.

A nonlocal quadratic functional of weighted differences is examined. The weights are based on image features and represent the affinity between different pixels in the image. By prescribing different formulas for the weights, one can generalize many local and nonlocal linear denoising algorithms, including the nonlocal means filter and the bilateral filter. In this framework one can easily show that continuous iterations of the generalized filter obey certain global characteristics and converge to a constant solution. The linear operator associated with the Euler-Lagrange equation of the functional is closely related to the graph Laplacian. We can thus interpret the steepest descent for minimizing the functional as a nonlocal diffusion process. This formulation allows a convenient framework for nonlocal variational minimizations, including variational denoising, Bregman iterations, and the recently proposed inverse scale space. It is also demonstrated how the steepest descent flow can be used for segmentation. Following kernel based methods in machine learning, the generalized diffusion process is used to propagate sporadic initial user's information to the entire image. Unlike classical variational segmentation methods, the process is not explicitly based on a curve length energy and thus can cope well with highly nonconvex shapes and corners. Reasonable robustness to noise is still achieved. Introduction.Evolutions based on partial differential equations (PDEs) have shown to provide very effective tools in image processing and computer vision. For some recent theory and applications see [3,47,46,19] and the references therein. Here we will try to give a unified approach to both denoising and segmentation tasks using nonlocal functionals and their respective nonlocal evolutions. In this paper we focus on the simpler case of quadratic functionals and linear evolutions.This study relates to many image processing disciplines and mathematical methods, some of which are not necessarily related to PDEs: spectral graph theory [20,42], segmentation by seeded region growing [1,65], graph-based segmentation [52,48,60], the Beltrami flow on Riemannian manifolds [34,54,33], relations between the graph Laplacian and the Laplace-Beltrami and other operators [6,44], and more.More specifically, the study was inspired by some recent studies on diffusion geometries [21,44,56], denoising by nonlocal means [10], and interactive segmentation [8,50,38].We summarize below only the most relevant results which will be used later in the paper.

The linear and nonlinear scale spaces, generated by the inherently real-valued diffusion equation, are generalized to complex diffusion processes, by incorporating the free Schrödinger equation. A fundamental solution for the linear case of the complex diffusion equation is developed. Analysis of its behavior shows that the generalized diffusion process combines properties of both forward and inverse diffusion. We prove that the imaginary part is a smoothed second derivative, scaled by time, when the complex diffusion coefficient approaches the real axis. Based on this observation, we develop two examples of nonlinear complex processes, useful in image processing: a regularized shock filter for image enhancement and a ramp preserving denoising process.

Signal and image enhancement is considered in the context of a new type of diffusion process that simultaneously enhances, sharpens, and denoises images. The nonlinear diffusion coefficient is locally adjusted according to image features such as edges, textures, and moments. As such, it can switch the diffusion process from a forward to a backward (inverse) mode according to a given set of criteria. This results in a forward-and-backward (FAB) adaptive diffusion process that enhances features while locally denoising smoother segments of the signal or image. The proposed method, using the FAB process, is applied in a super-resolution scheme. The FAB method is further generalized for color processing via the Beltrami flow, by adaptively modifying the structure tensor that controls the nonlinear diffusion process. The proposed structure tensor is neither positive definite nor negative, and switches between these states according to image features. This results in a forward-and-backward diffusion flow where different regions of the image are either forward or backward diffused according to the local geometry within a neighborhood.