Adaptive Subspace-Based Inverse Projections via Division Into Multiple Sub-Problems for Missing Image Data Restoration

Abstract

Abstract-This paper presents adaptive subspace-based inverse projections via division into multiple sub-problems (ASIP-DIMS) for missing image data restoration. In the proposed method, a target problem for estimating missing image data is divided into multiple sub-problems, and each sub-problem is iteratively solved with constraints of other known image data. By projection into a subspace model of image patches, the solution of each subproblem is calculated, where we call this procedure "subspacebased inverse projection" for simplicity. The proposed method can use higher-dimensional subspaces for finding unique solutions in each sub-problem, and successful restoration becomes feasible since a high level of image representation performance can be preserved. This is the biggest contribution of this paper. Furthermore, the proposed method generates several subspaces from known training examples and enables derivation of a new criterion in the above framework to adaptively select the optimal subspace for each target patch. In this way, the proposed method realizes missing image data restoration using ASIP-DIMS. Since our method can estimate any kind of missing image data, its potential in two image restoration tasks, image inpainting and super-resolution, based on several methods for multivariate analysis is also shown in this paper.

Multiple Subspace Model and Image-Inpainting Algorithm Based on Multiple Matrix Rank Minimization

“…In this paper, we assume that the missing region is provided as prior information. Various algorithms have been proposed for image inpainting such as exemplar based approaches [2]- [5], back projection approaches [6], [7], partial differential equation based approaches [8], [9] and deep learning based approaches [10], [11]. Exemplar based approaches recover pixels that are missing from an image (hereinafter referred to as the observed image) using either the observed image itself or a database of known images, with restoration achieved by finding a similar patch of pixels in an undamaged part of one of these images.…”

“…Back projection approaches assume that the complete image is represented by a linear combination of a few bases given by principal component analysis (PCA), and the missing pixels are restored using the coefficients of this linear combination. Ogawa et al proposed an adaptive subspace approach that divides the image patch into multiple clusters and performs a nonlinear inverse projection for low dimensional space using kernel PCA for each cluster [7]. This approach can achieve high quality restoration if the patches in the observed image are completely undamaged and if, moreover, the clusters are adequately studied.…”

“…This approach can achieve high quality restoration if the patches in the observed image are completely undamaged and if, moreover, the clusters are adequately studied. In this paper, however, we assume that the observed image is missing more than half of its pixels, meaning that the problem setting of this paper differs from that of [7]. Deep learning based approaches have evolved in recent years [10], [11] and have achieved remarkable inpainting results, such as restoring a partly hidden face.…”