Deformable strokes towards temporally coherent video painting

Abstract

We present an automatic and robust technique for creating non-photorealistic rendering (NPR) and animation, starting from a video that depicts the shape details and follows the motion of underlying objects. We generate NPR from the initial frame of the source video using a greedy algorithm for stroke placements and models, in combination with a saliency map and a flow-guided difference-of-Gaussian filter. Our stroke model uses a set of triangles whose vertices are particles and whose edges are springs. Using a physicsbased framework, the generated and rendered strokes are translated, rotated and deformed by forces exerted from the sequential frames. External forces acting on strokes are calculated according to temporally and spatially smoothed per-pixel optical flow vectors. After simulating each frame, we delete unnecessary strokes and add new strokes for disappearing and appearing objects, but only if necessary to avoid popping and scintillation. Our framework automatically generates the coherent animation of rendered strokes, preserving the appearance details and animating strokes along with the underlying objects. This had been difficult to achieve with previous user-guided methods and automatic but limited transformations methods.

Facial Feature Model for a Portrait Video Stylization

“…For subregion 3, following the method in [33], strokes were placed and oriented based on the saliency map and tangent flows, before we performed the texture mapping on them. However, for subregion 1, the difference from [33] is that we located the smallest modeled strokes based only on the tangent flow. In subregion 2, we colored the facial feature area in the rendered image with the same pixel colors of the areas from the source image.…”

“…Second, based on the extracted saliency map and tangent flows from the sequences, the render started with the first frame of the source video through different rendering methods on the sub-regions. For subregion 3, following the method in [33], strokes were placed and oriented based on the saliency map and tangent flows, before we performed the texture mapping on them. However, for subregion 1, the difference from [33] is that we located the smallest modeled strokes based only on the tangent flow.…”

“…To generate an animation from a portrait video, many stroke-based methods have been created. The method in [9] was an early development for video painting, where they conducted the video animation by translating and rotating the strokes within its limitations; the method in [15] treated the strokes as points and used the optical flow for animation, and Krompiec, P. [33] presented a good method that could animate the strokes along with the underlying objects; however, for rendering small faces in videos, just keeping different sizes of the strokes to maintain the distinct features of the face is not enough.…”

“…Thus, it is of much significance to properly design and initialize the strokes in the first frame. Following [33], we used the stroke model shown in Figure 5a for the non-facial area to follow the underlying motion coherently during the video sequences and keep the least number of strokes. This model is composed of six particles and nine springs for the deformation and motion between frames.…”

“…After rendering the first frame, we started to animate the video sequences. Unlike most video stylization methods that only focus on the translation or orientation of moving strokes, we also performed deformations on strokes based on our previous work [33]. The exact calculation of the position and deformation of each stroke in the next frame was based on the physics-based framework [35], and the motion estimation method [36].…”