Clustering Dynamic Textures with the Hierarchical EM Algorithm for Modeling Video

Abstract

Dynamic texture (DT) is a probabilistic generative model, defined over space and time, that represents a video as the output of a linear dynamical system (LDS). The DT model has been applied to a wide variety of computer vision problems, such as motion segmentation, motion classification, and video registration. In this paper, we derive a new algorithm for clustering DT models that is based on the hierarchical EM algorithm. The proposed clustering algorithm is capable of both clustering DTs and learning novel DT cluster centers that are representative of the cluster members in a manner that is consistent with the underlying generative probabilistic model of the DT. We also derive an efficient recursive algorithm for sensitivity analysis of the discrete-time Kalman smoothing filter, which is used as the basis for computing expectations in the E-step of the HEM algorithm. Finally, we demonstrate the efficacy of the clustering algorithm on several applications in motion analysis, including hierarchical motion clustering, semantic motion annotation, and learning bag-of-systems (BoS) codebooks for dynamic texture recognition.

Joint Motion Segmentation and Background Estimation in Dynamic Scenes

“…In addition, most methods require prior knowledge in the form of a "clean" training video containing only the background. Dynamic texture models have also shown promise in clustering the microscopic and macroscopic motion patterns present in dynamic scenes [8][9][10]. [8] performs motion segmentation by clustering video patches using a mixture of DTs.…”

A Scalable and Accurate Descriptor for Dynamic Textures Using Bag of System Trees

“…An alternative approach, presented in [2], [3], is based on the probabilistic framework of the DT. For each video, spatiotemporal patches are extracted using dense sampling, and a dynamic texture mixture (DTM) is learned for each video using the EM algorithm [27].…”

“…T HE bag-of-systems (BoS) representation [1], a high-level descriptor of motion in a video, has seen promising results in video texture classification [2], [3], [4]. The BoS representation of videos is analogous to the bag-of-words representation of text documents, where documents are represented by counting the occurrences of each word, or the bag-of-visual-words representation of images, where images are represented by counting the occurrences of visual codewords in the image.…”

“…For even modest sized codebooks, this results in a heavy computational load. For example, the BoS codebooks of [2] are limited to only 8 codewords and [3] uses a maximum of 64 codewords.…”

On the Segmentation and Classification of Water in Videos

“…A second dominant approach is the global modeling of videos using Linear Dynamical Systems (LDS) (Chan and Vasconcelos, 2008;Doretto et al, 2003;Mumtaz et al, 2013;Saisan et al, 2001). In its essence, LDS is a latent variable model which projects video frames to a lower dimensional space and tracks the temporal behaviour in that lower dimensional space.…”