Onion-Peel Networks for Deep Video Completion

Abstract

We propose the onion-peel networks for video completion. Given a set of reference images and a target image with holes, our network fills the hole by referring the contents in the reference images. Our onion-peel network progressively fills the hole from the hole boundary enabling it to exploit richer contextual information for the missing regions every step. Given a sufficient number of recurrences, even a large hole can be inpainted successfully. To attend to the missing information visible in the reference images, we propose an asymmetric attention block that computes similarities between the hole boundary pixels in the target and the non-hole pixels in the references in a non-local manner. With our attention block, our network can have an unlimited spatial-temporal window size and fill the holes with globally coherent contents. In addition, our framework is applicable to the image completion guided by the reference images without any modification, which is difficult to do with the previous methods. We validate that our method produces visually pleasing image and video inpainting results in realistic test cases.1 [t-6, t-3, t-1, t+3, t+6], where the current frame is at t arXiv:1908.08718v1 [cs.CV]

CIRO: The Effects of Visually Diminished Real Objects on Human Perception in Handheld Augmented Reality

“…Note that, however, the CIRO we consider in this paper, a pedestrian, in most cases simply passes by in the background. Additionally, note that obviously, compared to the ground truth (perfect removal and inpainting), any systems, including ours, are bound to exhibit some noticeable artifacts [52,56]. Figure 5 shows the worst-case scenario of visual artifacts captured in our inpainting system.…”

“…We compared and evaluated the inpainting performance of the original VINet [52] and our lightweight, real-time version using the structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) [55,56]. SSIM measures the similarity in structure between the two images, and PSNR, the image's distortion.…”

Video Inpainting by Frame Alignment with Deformable Convolution

“…To avoid the difficult flow estimation and efficiently obtain a large temporal window, Onion-Peel Networks (OP-Net) [6] employs a spatio-temporal attention module in the deep feature space, which can search the input video for applicable patches in a non-local manner. It has great capability to fill never visible areas by borrowing patches from visible regions.…”

“…In parallel with our research, other methods [28]- [30] have been proposed. Zeng et al [29] employed non-local attention modules like OPNet [6]. Liu et al [30] used only deformable convolutions for frame alignment.…”

“…From the Places2 dataset [38], we randomly select an image and cropped it into a square five times. We change the window size, angle, and position for every cropping to produce various motions as in CPNet [5] and OPNet [6].…”