Abstract. We propose an image representation scheme combining the local and nonlocal characterization of patches in an image. Our representation scheme can be shown to be equivalent to a tight frame constructed from convolving local bases (e.g. wavelet frames, discrete cosine transforms, etc.) with nonlocal bases (e.g. spectral basis induced by nonlinear dimension reduction on patches), and we call the resulting frame elements convolution framelets. Insight gained from analyzing the proposed representation leads to a novel interpretation of a recent highperformance patch-based image inpainting algorithm using Point Integral Method (PIM) and Low Dimension Manifold Model (LDMM) [Osher, Shi and Zhu, 2016]. In particular, we show that LDMM is a weighted 2 -regularization on the coefficients obtained by decomposing images into linear combinations of convolution framelets; based on this understanding, we extend the original LDMM to a reweighted version that yields further improved inpainting results. In addition, we establish the energy concentration property of convolution framelet coefficients for the setting where the local basis is constructed from a given nonlocal basis via a linear reconstruction framework; a generalization of this framework to unions of local embeddings can provide a natural setting for interpreting BM3D, one of the state-of-the-art image denoising algorithms.Key words. image patches, convolution framelets, regularization, nonlocal methods, inpainting AMS subject classifications. 68U10, 68Q251. Introduction. In the past decades, patch-based techniques such as Non-Local Means (NLM) and Block-Matching with 3-D Collaborative Filtering (BM3D) have been successfully applied to image denoising and other image processing tasks [7,8,16,11,64,30]. These methods can be viewed as instances of graph-based adaptive filtering, with similarity between pixels determined not solely by their pixel values or spatial adjacency, but also by the (weighted) 2 -distance between their neighborhoods, or patches containing them. The effectiveness of patch-based algorithms can be understood from several different angles. On the one hand, patches from an image often enjoy sparse representations with respect to certain redundant families of vectors, or unions of bases, which motivated several dictionaryand sparsity-based approaches [21,10,41]; on the other hand, the nonlocal characteristics of patch-based methods can be used to build highly data-adaptive representations, accounting for nonlinear and self-similar structures in the space of image patches [26,34]. Combined with adaptive thresholding, these constructions have connections to classical wavelet-based and total variation algorithms [63,47]. Additionally, the patch representation of signals has specific structures that can be exploited in regularization; for example, the inpainting algorithm ALOHA [28] utilized the low-rank block Hankel structure of certain matrix representation of image patches.Among the many theoretical frameworks built to understand these patch-b...

No abstract