Depth-Guided Disocclusion Inpainting of Synthesized RGB-D Images

Abstract

We propose to tackle the problem of RGB-D image disocclusion inpainting when synthesizing new views of a scene by changing its viewpoint. Indeed, such a process creates holes both in depth and color images. First, we propose a novel algorithm to perform depth-map disocclusion inpainting. Our intuitive approach works particularly well for recovering the lost structures of the objects and to inpaint the depth-map in a geometrically plausible manner. Then, we propose a depth-guided patch-based inpainting method to fill-in the color image. Depth information coming from the reconstructed depth-map is added to each key step of the classical patch-based algorithm from Criminisi et al. in an intuitive manner. Relevant comparisons to the state-of-the-art inpainting methods for the disocclusion inpainting of both depth and color images are provided and illustrate the effectiveness of our proposed algorithms.

Guidefill: GPU Accelerated, Artist Guided Geometric Inpainting for 3D Conversion of Film

“…The first pipeline assumes that each frame of video is accompanied by a depth map (and hence is more applicable to footage from RGB-D cameras). The new viewpoint is generated by "warping" the original footage based on the given depth map and known or estimated camera parameters -see [10] for an excellent recent overview. This pipeline has applications including 3D TV and free-viewpoint rendering [48,19].…”

“…This step is considered one of the most technical and time consuming pieces of the pipeline [40]. However, while the disocclusion step arising in the first pipeline has received a lot of attention in the literature, see for example [10,46,45,19,48,24,34,17,26,30] to name a few, the disocclusion step arising in the second pipeline relevant to film has received far less attention. We are, in fact, to the best of our knowledge the first paper to address it directly.…”

“…Firstly, while the methods above need to inpaint both the color information at the new view and the corresponding new depth map -we get the depth map at the new viewpoint for free. This is important because most of the methods in the literature either devote quite a bit of effort to inpainting the depth map [10], or else do so based on rough heuristics [46,45,19,48,24,34,17,26,30], which, as noted in [10, Sec. II.C.…”

“…This means that we unlike the above approaches always know which pixels to use for inpainting and do not have have to worry about (for example) inpainting a piece of the foreground into the background. By contrast, all of the above methods have to rely on imperfect heuristics to guess based on the depth map which pixels belong to which object -see [10,Sec. II.B.…”

“…These methods are so basic that they do not appear to inpaint the depth map. In the midrange you have a variety of methods based on first inpainting the depth map, and then applying a depth aided variant to Criminisi's method -examples include [45,46,17,26,30,10], see also [10] for an overview of the state of the art. Unfortunately, until recently most of these approaches have been limited in the sense that too little attention has been given to the depth inpainting step, which is done based on crude heuristics, while the most of the attention is given to the subsequent color inpainting step using a depthaided variant of Criminisi.…”