Unsupervised Discovery of Temporal Structure in Noisy Data with Dynamical Components Analysis

Abstract

Linear dimensionality reduction methods are commonly used to extract lowdimensional structure from high-dimensional data. However, popular methods disregard temporal structure, rendering them prone to extracting noise rather than meaningful dynamics when applied to time series data. At the same time, many successful unsupervised learning methods for temporal, sequential and spatial data extract features which are predictive of their surrounding context. Combining these approaches, we introduce Dynamical Components Analysis (DCA), a linear dimensionality reduction method which discovers a subspace of high-dimensional time series data with maximal predictive information, defined as the mutual information between the past and future. We test DCA on synthetic examples and demonstrate its superior ability to extract dynamical structure compared to commonly used linear methods. We also apply DCA to several real-world datasets, showing that the dimensions extracted by DCA are more useful than those extracted by other methods for predicting future states and decoding auxiliary variables. Overall, DCA robustly extracts dynamical structure in noisy, high-dimensional data while retaining the computational efficiency and geometric interpretability of linear dimensionality reduction methods.DCA code is available at: https://github.com/BouchardLab/DynamicalComponentsAnalysis Preprint. Under review.

Pareto-Optimal Data Compression for Binary Classification Tasks

“…The general case remains unsolved, and is obviously even harder than the above-mentioned 1-vector autoencoding problem. The recent work [6,7] review the state-of-the art as well as presenting Contrastive Predictive Coding and Dynamic Component Analysis, powerful new distillation techniques for time series, following the long tradition of setting f = g even though this is provably not optimal for the Gaussian case as shown in [8].…”

“…While fine-grained binning has no effect on the entropy H(Y ) and negligible effect on I(W, Y ), it dramatically reduces the entropy of our data. Whereas H(W ) = ∞ since W is continuous 7 , H(W ) = log N is finite, approaching infinity only in the limit of infinitely many infinitesimal bins. Taken together, these scalings of I and H imply that the leftmost part of the Pareto frontier I * (H * ), defined by equation ( 1) and illustrated in Figure 1, asymptotes to a horizontal line of height I * = I(X, Y ) as H * → ∞.…”

Multivariate temporal process monitoring with graph‐based predictable feature analysis

“…Alternatively, since the goal of GPFA is to maximize the predictability of their DLVs, we adopt the predictive information instead of the evaluation index. [33,37] Given a temporal series X ¼ x t f g n t¼1 , x t ℝ m with its probability density function (pdf) denoted as P(X), define…”

“…In GPFA, the weight matrix W is obtained when the predictive information in the latent scores s t = W > x t is maximized, which coincides with the definition of PI pred T S ð Þ. To obtain the explicit expression of PI pred T S ð Þ, inspired by the denotations in Clark et al, [33] a spatiotemporal covariance matrix Σ T (X) is defined to encode all second-order statistics of X across T time steps.…”

“…It is noted that parameter selection of GPFA is an open problem. Since GPFA takes maximal predictability as the objective when trying to extract dynamic features, information theory, especially entropy, [33] is an effective technique to quantify the predictability. Therefore, an index termed predictive information [34] is adopted in this work to optimize the parameters of GPFA.…”