High-Quality Real-Time Video Inpaintingwith PixMix

Abstract

While image inpainting has recently become widely available in image manipulation tools, existing approaches to video inpainting typically do not even achieve interactive frame rates yet as they are highly computationally expensive. Further, they either apply severe restrictions on the movement of the camera or do not provide a high-quality coherent video stream. In this paper we will present our approach to high-quality real-time capable image and video inpainting. Our PixMix approach even allows for the manipulation of live video streams, providing the basis for real Diminished Reality (DR) applications. We will show how our approach generates coherent video streams dealing with quite heterogeneous background environments and non-trivial camera movements, even applying constraints in real-time.

A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects

“…On the other hand, this technique can eliminate obstacles with no background (e.g., drawings on a wall and manholes on a street). Most of existing work assumes that the background is planar [9,10,21] because image inpainting is potentially a screen space image processing. The recent work can handle curved [22] or multiple surfaces [23].…”

“…Homography transformations of images [13,19,35,36,41,42], rendering texture 3D models [6,31,43], rendering voxels [37], and image-based rendering (IBR) [25,26,30,44] are the major approaches. In inpaint, pixels or image patches are synthesized in the ROI to fill in the region with plausible pixels [9,10,21,38]. 5.…”

“…Enomoto et al used fiducial markers, ARTag [54], for registering all cameras in a unified coordinate system, and therefore, each camera can move in 6DoF while the marker is visible in the view [41]. Herling and Broll used an object tracker that continuously detects and tracks the ROI to fill in the ROI with pixels using inpaint [9,10]. If the target object to be diminished is a marker [21,22,38] or a marker-attached object [31], then the fiducial markers are used to estimate the camera pose.…”

“…In inpaint methods, an object of interest is tracked frame-by-frame [9,10,23]. Herling and Broll first used classic Snake algorithm [63] to track contours of an object of interest [9] and improved their ROI detection and tracking algorithms for video usages [10].…”

“…In augmented reality, real-virtual inconsistency mainly appears where real and virtual objects contact each other, i.e., geometric registration of virtual objects is one of the most important issues. From this point of view, in DR, real-virtual borders will appear all around the virtual regions since real regions surround the virtual ones methodology, although image inpainting had been considered a computationally expensive process to be run in real time because the method repetitively searches and composes image patches to fill in missing regions [9,10].…”