Synthesizing Dynamic Patterns by Spatial-Temporal Generative ConvNet

Abstract

Video sequences contain rich dynamic patterns, such as dynamic texture patterns that exhibit stationarity in the temporal domain, and action patterns that are non-stationary in either spatial or temporal domain. We show that a spatialtemporal generative ConvNet can be used to model and synthesize dynamic patterns. The model defines a probability distribution on the video sequence, and the log probability is defined by a spatial-temporal ConvNet that consists of multiple layers of spatial-temporal filters to capture spatialtemporal patterns of different scales. The model can be learned from the training video sequences by an "analysis by synthesis" learning algorithm that iterates the following two steps.Step 1 synthesizes video sequences from the currently learned model. Step 2 then updates the model parameters based on the difference between the synthesized video sequences and the observed training sequences. We show that the learning algorithm can synthesize realistic dynamic patterns.

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns

“…[22] did not work on dynamic patterns such as those in the video sequences. [39] is a generalization of [22] for dynamic patterns by adopting spatial-temporal ConvNets [3] to capture spatial and temporal features of the video sequences. Recently, [40] proposed a volumetric version of the energy-based generative ConvNet for modeling 3D shape patterns.…”

“…Recently, [40] proposed a volumetric version of the energy-based generative ConvNet for modeling 3D shape patterns. This paper is an expanded version of our conference paper in [39].…”

Representation Learning: A Statistical Perspective

“…Similar to GAN, we can pair the generator model with an energy-based model (Ngiam et al, 2011;Dai et al, 2014;Lu et al, 2016;Xie et al, 2016Xie et al, , 2017Xie et al, , 2018cGao et al, 2018a), instead of a discriminator model. Similar to the discriminator model, the energy-based model is also defined by a bottom-up network.…”

Learning Dynamic Generator Model by Alternating Back-Propagation through Time

“…Models based on spatial-temporal filters or kernels. The patterns in the video data can also be modeled by spatialtemporal filters by treating the data as 3D (2 spatial dimensions and 1 temporal dimension), such as a 3D energy-based model (Xie, Zhu, and Wu 2017) where the energy function is parametrized by a 3D bottom-up ConvNet, or a 3D generator model (Han et al 2019) where a top-down 3D Con-vNet maps a latent random vector to the observed video data. Such models do not have a dynamic structure defined by a transition model, and they are not convenient for predicting future frames.…”