Video Primal Sketch: A Unified Middle-Level Representation for Video

Abstract

This paper presents a middle-level video representation named video primal sketch (VPS), which integrates two regimes of models: (i) sparse coding model using static or moving primitives to explicitly represent moving corners, lines, feature points, etc., (ii) FRAME /MRF model reproducing feature statistics extracted from input video to implicitly represent textured motion, such as water and fire. The feature statistics include histograms of spatio-temporal filters and velocity distributions. This paper makes three contributions to the literature: (i) Learning a dictionary of video primitives using parametric generative models; (ii) Proposing the spatio-temporal FRAME and motion-appearance FRAME models for modeling and synthesizing textured motion; and (iii) Developing a parsimonious hybrid model for generic video representation. Given an input video, VPS selects the proper models automatically for different motion patterns and is compatible with high-level action representations. In the experiments, we synthesize a number of textured motion; reconstruct real videos using the VPS; report a series of human perception experiments to verify the quality of reconstructed videos; demonstrate how the VPS changes over the

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns

“…Dynamic textures. Generating dynamic textures or textured motions have been studied by [1], [2], [41], [42]. For instance, [1] proposed a vector auto-regressive model coupled with framewise dimension reduction by single value decomposition.…”

Motion-Based Generator Model: Unsupervised Disentanglement of Appearance, Trackable and Intrackable Motions in Dynamic Patterns

“…Intrackability (or trackability) is an important concept of motion patterns, which has been studied in (Gong and Zhu 2012). It was demonstrated in (Wu, Zhu, and Guo 2008;Gong and Zhu 2012;Han, Xu, and Zhu 2015) that trackability changes over scales, densities, and stochasticity of (1)…”

“…Intrackability (or trackability) is an important concept of motion patterns, which has been studied in (Gong and Zhu 2012). It was demonstrated in (Wu, Zhu, and Guo 2008;Gong and Zhu 2012;Han, Xu, and Zhu 2015) that trackability changes over scales, densities, and stochasticity of the dynamics. For example, trackability of a video of waterfall will depend on the distance between the observed target The penalty parameter for smoothness is 0.005. and the observer.…”

Synthesizing Dynamic Patterns by Spatial-Temporal Generative ConvNet

“…Dynamic textures or textured motions have been studied by [2,24,25,9]. For instance, [2] proposed a vector auto-regressive model coupled with frame-wise dimension reduction by single value decomposition.…”