Categorizing Dynamic Textures Using a Bag of Dynamical Systems

Abstract

We consider the problem of categorizing video sequences of dynamic textures, i.e., nonrigid dynamical objects such as fire, water, steam, flags, etc. This problem is extremely challenging because the shape and appearance of a dynamic texture continuously change as a function of time. State-of-the-art dynamic texture categorization methods have been successful at classifying videos taken from the same viewpoint and scale by using a Linear Dynamical System (LDS) to model each video, and using distances or kernels in the space of LDSs to classify the videos. However, these methods perform poorly when the video sequences are taken under a different viewpoint or scale. In this paper, we propose a novel dynamic texture categorization framework that can handle such changes. We model each video sequence with a collection of LDSs, each one describing a small spatiotemporal patch extracted from the video. This Bag-of-Systems (BoS) representation is analogous to the Bag-of-Features (BoF) representation for object recognition, except that we use LDSs as feature descriptors. This choice poses several technical challenges in adopting the traditional BoF approach. Most notably, the space of LDSs is not euclidean; hence, novel methods for clustering LDSs and computing codewords of LDSs need to be developed. We propose a framework that makes use of nonlinear dimensionality reduction and clustering techniques combined with the Martin distance for LDSs to tackle these issues. Our experiments compare the proposed BoS approach to existing dynamic texture categorization methods and show that it can be used for recognizing dynamic textures in challenging scenarios which could not be handled by existing methods.

Recognizing Activities via Bag of Words for Attribute Dynamics

“…This is related to the bag-of-systems framework of [20,1], where a set of dynamic textures (DTs) [5] were used to characterize dynamic scenes. The main challenge of this dictionary leaning problem is the difficulty of identifying the "centroid" of a collection of dynamic textures, due to the non-Euclidean nature of the space of linear dynamic systems.…”

“…The main challenge of this dictionary leaning problem is the difficulty of identifying the "centroid" of a collection of dynamic textures, due to the non-Euclidean nature of the space of linear dynamic systems. [20] bypasses this problem with resort to a somewhat heuristic combination of multi-dimensional scaling and kmeans (denoted MDS-kM); while [1] presents a procedure to directly average dynamic models in the parameter space, the approach only works for LDS's. We propose an alternative principled solution, which is specifically designed for clustering attribute sequences, and has a number of advantages over MDS-kM.…”

Initial-state invariant Binet-Cauchy kernels for the comparison of Linear Dynamical Systems

“…This is because we cannot compute an explicit embedding for HOOF time-series and hence we cannot use LDS system parameters and therefore the JCF that is required to compute the Binet-Cauchy maximum singular value kernel.) will use boxplots to show these statistics for each choice of metric and histogram kernel for a range of bin sizes (20-100) and system orders (1)(2)(3)(4)(5)(6)(7)(8)(9)(10)(11)(12)(13)(14)(15)(16)(17)(18)(19)(20). Figure 1 shows a generic box-plot.…”

“…For instance, [4] uses LDSs to model surgical gestures in video data from the DaVinci robot; [5], [6] use LDSs to model the appearance of a deforming heart in a magnetic resonance image sequence; [7], [8], [9], [10], [11], [12], [13] use LDSs to model the appearance of dynamic textures, such as water or fire, in a video sequence; [14], [15], [16], [17], [18], [19] use LDSs to model human gaits, such as walking or running, in motion capture and video data; [20] uses LDSs to model the appearance of moving faces; and [21] uses LDSs to model audio-visual lip articulations.…”

Convolutional Recognition of Dynamic Textures With Preliminary Categorization

“…In past decades, a variety of different approaches have been proposed for recognition of the DTs, such as the Linear Dynamic System (LDS) methods (Ravichandran et al, 2013), GIST method (Oliva and Torralba, 2001), the Local Binary Pattern (LBP) methods (Zhao and Pietikainen, 2007a), wavelet methods (Dubois et al, 2009;Dubois et al, 2015), morphological methods (Dubois et al, 2012), deep multilayer networks Arashloo et al, 2017), among others.…”