Joint texture-depth pixel inpainting of disocclusion holes in virtual view synthesis

Abstract

Abstract-Transmitting texture and depth maps from one or more reference views enables a user to freely choose virtual viewpoints from which to synthesize images for observation via depth-image-based rendering (DIBR). In each DIBR-synthesized image, however, there remain disocclusion holes with missing pixels corresponding to spatial regions occluded from view in the reference images. To complete these holes, unlike previous schemes that rely heavily (and unrealistically) on the availability of a high-quality depth map in the virtual view for inpainting of the corresponding texture map, in this paper a new Joint TextureDepth Inpainting (JTDI) algorithm is proposed that simultaneously fill in missing texture and depth pixels. Specifically, we first use available partial depth information to compute priority terms to identify the next target pixel patch in a disocclusion hole for inpainting. Then, after identifying the best-matched texture patch in the known pixel region via template matching for texture inpainting, the variance of the corresponding depth patch is copied to the target depth patch for depth inpainting. Experimental results show that JTDI outperforms two previous inpainting schemes that either does not use available depth information during inpainting, or depends on the availability of a good depth map at the virtual view for good inpainting performance.

Disocclusion hole-filling in DIBR-synthesized images using multi-scale template matching

“…For each sequence, DIBR has been used to generate the reference view #3 using texture and the disparity map of view #1. To quantitatively and qualitatively evaluate the performance of proposed algorithm, the generated view #3 was inpainted using a patch-size of 9 × 9 pixels and compared with Criminisi [3] and JTDI [7] methods. Both the comparators, Criminisi [3] and JTDI [7] employs single-scale exhaustive TM to find C P for filling disocclusion holes.…”

“…To quantitatively and qualitatively evaluate the performance of proposed algorithm, the generated view #3 was inpainted using a patch-size of 9 × 9 pixels and compared with Criminisi [3] and JTDI [7] methods. Both the comparators, Criminisi [3] and JTDI [7] employs single-scale exhaustive TM to find C P for filling disocclusion holes. For numerical analysis, the original view #3 of image datasets was used as the ground truth for all peak signal-to-noise ratio (PSNR) calculations, with the PSNR computed for both the whole image and hole regions.…”

“…In contrast, we perform joint texture and depth pixel filling of disocclusion holes as done in [7], but employ multi-scale 1) Depth Layer Selection and Hole-filling: First, we select the target patch P T and the corresponding depth target patch D T . The order of selecting target patches for filling is very important, but is outside the scope of this paper; we will simply use the P T selection method in [7]. The known values of D T are used to determine the mean of depth values d mean .…”

“…1. For disocclusion hole-filling, a recent Joint Texture and Depth Inpainting (JTDI) algorithm [7] fills texture and depth hole pixels alternately: use available depth information to fill in textural pixel holes, then use inpainted textural information to fill corresponding depth pixel holes. However, JTDI still employs full-image TM, which is computation-expensive.…”

“…In the second category are nonlocal schemes such as Criminisi's template-matching (TM) [3] algorithm, that fill in missing pixels in a target region by identifying similar pixel patterns in faraway known region, assuming the well recognized selfsimilarity characteristic commonly observed in natural images. Recently, TM has been adopted for disocclusion hole-filling in DIBR-synthesized images as well [6], [7]. There remain two problems.…”