Multi-View Inpainting for Image-Based Scene Editing and Rendering

Abstract

We propose a method to remove objects such as people and cars from multi-view urban image datasets, enabling free-viewpoint Image-Based Rendering (IBR) in the edited scenes. Our method combines information from multi-view 3D reconstruction with image inpainting techniques, by formulating the problem as an optimization of a global patch-based objective function. We use IBR techniques to reproject information from neighboring views, and 3D multi-view stereo reconstruction to perform multi-view coherent initialization for inpainting of pixels not filled by reprojection. Our algorithm performs multi-view consistent inpainting for color and 3D by blending reprojections with patch-based image inpainting. We run our algorithm on casually captured datasets, and Google Street View data, removing objects such as cars, people and pillars, showing that our approach produces results of sufficient quality for free-viewpoint IBR on "cleaned up" scenes, as well as IBR scene editing, such as limited displacement of real objects.

Plane-based multi-view inpainting for image-based rendering in large scenes

“…The resulting depthmaps are thus not adapted to a free-viewpoint IBR context. Thonat et al [2016] introduce the first method for multi-view inpainting with output suitable for free-viewpoint IBR. Their approach imposes soft multi-view coherence while inpainting separately in each input image.…”

“…For 3D graphics applications, it is often necessary or convenient to remove objects from the scene, e.g., parked cars or furniture in a room. This can be achieved by multi-view inpainting, i.e., by removing the undesired objects in each input image and filling corresponding pixels and depth using inpainting or completion [Huang et al 2014;Thonat et al 2016]. Single-image inpainting methods [Bertalmio et al 2000;Criminisi et al 2004] are not designed to treat multi-view datasets even when they handle effects such as perspective [Huang et al 2014].…”

“…Single-image inpainting methods [Bertalmio et al 2000;Criminisi et al 2004] are not designed to treat multi-view datasets even when they handle effects such as perspective [Huang et al 2014]. Recent work provides initial solutions to this problem [Baek et al 2016;Thonat et al 2016], but suffers from four limitations: 1) multi-view coherence is applied progressively across neighboring images and is often inaccurate or incomplete, 2) perspective effects are not correctly reproduced during inpainting, resulting in visual artifacts and blurring, 3) the quality of depth synthesis is insufficient and 4) the methods are not designed to handle large datasets, since they often use expensive algorithmic solutions operating on all images in the dataset. We target scenes containing man-made structures, corresponding to city blocks or apartments, containing up to hundreds of input images.…”

Multi-view Inpainting for RGB-D Sequence

“…Baek et al [18] present a multi-view based method to complete the userdefined region by jointly inpaint the color and depth image, which takes advantages from SfM to achieve geometric registration among different views. Similarly, Thonat et al [10] enable free-viewpoint image based rendering with reprojected information from neighboring views. Also a refined method is proposed in the following work [11] that performs inpainting on intermediate, local planes in order to preserve perspective as well as to ensure multi-view coherence.…”

Fast Light Field Inpainting Propagation Using Angular Warping and Color-Guided Disparity Interpolation

“…Although the problem of image inpainting has already been extensively studied, only a few methods in the literature address the specific case of 4D light fields. One can find some work on the related topic of multiview image inpainting, however the methods as in [1], [2] are designed for captures with large baselines and do not generalize well for dense light fields.…”