Video Painting via Motion Layer Manipulation

Abstract

Temporal coherence is an important problem in Non-Photorealistic Rendering for videos. In this paper, we present a novel approach to enhance temporal coherence in video painting. Instead of painting on video frame, our approach first partitions the video into multiple motion layers, and then places the brush strokes on the layers to generate the painted imagery. The extracted motion layers consist of one background layer and several object layers in each frame. Then, background layers from all the frames are aligned into a panoramic image, on which brush strokes are placed to paint the background in one-shot. The strokes used to paint object layers are propagated frame by frame using smooth transformations defined by thin plate splines. Once the background and object layers are painted, they are projected back to each frame and blent to form the final painting results. Thanks to painting a single image, our approach can completely eliminate the flickering in background, and temporal coherence on object layers is also significantly enhanced due to the smooth transformation over frames. Additionally, by controlling the painting strokes on different layers, our approach is easy to generate painted video with multi-style. Experimental results show that our approach is both robust and efficient to generate plausible video painting.

Global contrast based salient region detection

“…Figure 11. (Middle, right) FT [2] and RC saliency maps are used respectively for stylized rendering [16] of an input image (left). Our method produces a better saliency map, see insets, resulting in improved preservation of details, e.g., around the head and the fence regions.…”

“…Inspired by this observation, a number of non-photorealistic rendering (NPR) efforts use saliency maps to generate interesting effects [7]. We experimentally compared our work with the most related, state-of-the-art saliency detection algorithm [2] in the context of a recent NPR technique [16] (see Figure 11). Our RC-maps give better saliency masks, which help the NPR method to better preserve details in important image parts and region boundaries, while smoothing out others.…”

Deformable strokes towards temporally coherent video painting

“…A semantic driven approach, based on recent image parsing techniques in computer vision, benefits and leads to better simulations of artists' painting styles [8]. Huang et al used motion layer manipulations [9], and Collomosse et al [10] analyzed the video sequence as a spatiotemporal volume. O'Donovan et al proposed a method to use video segmentation and key-framed control strokes [11].…”

“…The resulting animations were insensitive to the motion of objects and the shape details. Recent studies [9][10][11], however, have rather focused on how to combine and utilize multiple image processing techniques and to apply image processing techniques at key-frames to guide the transformations of strokes. Despite acknowledged success, the current stroke-based rendering methods require a user's guidance at key-frames including picking key-frames, segmenting objects at key-frames, user-guidance at key-frames, and generating motion layers.…”

“…Among numerous video painterly rendering techniques [4][5][6][7][8][9][10][11][12][13][14][15][16][17][18], the major scheme uses a set of image processing methods that includes blur, edge detection, tangent flow, segmentation, feature extraction, saliency maps, and optical flows. Since most image processing methods are sensitive to illumination variations and noise in the source images, the quality of the resulting outputs depends on the quality of the image processing techniques.…”