Dynamic Mode Decomposition for Robust PCA with Applications to Foreground/Background Subtraction in Video Streams and Multi-Resolution Analysis

Abstract

Accurate and real-time video surveillance techniques for removing background variations in a video stream, which are highly correlated between frames, are at the forefront of modern data-analysis research. The objective in such algorithms is to highlight foreground objects of potential interest. Background/foreground separation is typically an integral step in detecting, identifying, tracking, and recognizing objects in video sequences. Most modern computer vision applications demand algorithms that can be implemented in real-time, and that are robust enough to handle diverse, complicated, and cluttered backgrounds. Competitive methods often need to be flexible enough to accommodate changes in a scene due to, for instance, illumination changes that can occur throughout the day, or location changes where the application is being implemented. Given the importance of this task, a variety of iterative techniques and methods have already been developed in order to perform background/foreground separation [4,8,11,15,23,24] (See also, for instance, the recent reviews by Bouwmans [2] and Benezeth et al. [1], which compare error and timing of various methods).One potential viewpoint of this computational task is as a matrix separation problem into low-rank (background) and sparse (foreground) components. Recently, this viewpoint has been advocated by Candès et al. in the framework of robust principal component analysis 19-1 (RPCA) [4]. By weighting a combination of the nuclear and the L 1 norms, a convenient convex optimization problem (principal component pursuit) was demonstrated, under suitable assumptions, to exactly recover the low-rank and sparse components of a given data-matrix (or video for our purposes). It was also compared to the state-of-the-art computer vision procedure developed by De La Torre and Black [10]. We advocate a similar matrix separation approach, but by using the method of dynamic mode decomposition (DMD) [5,17,[20][21][22]26] (see also Kutz [9] for a tutorial review). This method, which essentially implements a Fourier decomposition of correlated spatial activity of the video frames in time, distinguishes the stationary background from the dynamic foreground by differentiating between the nearzero Fourier modes and the remaining modes bounded away from the origin, respectively [7]. Originally introduced in the fluid mechanics community, DMD has emerged as a powerful tool for analyzing the dynamics of nonlinear systems [5,17,[20][21][22]26].In the application of video surveillance, the video frames can be thought of as snapshots of some underlying complex/nonlinear dynamics. The DMD decomposition yields oscillatory time components of the video frames that have contextual implications. Namely, those modes that are near the origin represent dynamics that are unchanging, or changing slowly, and can be interpreted as stationary background pixels, or low-rank components of the data matrix. In contrast, those modes bounded away from the origin are changing on O(1) timescales or faster, and represent ...

Modal Analysis of Fluid Flows: An Overview

“…Because DMD is rooted firmly in linear algebra, the method is highly extensible, spurring considerable algorithmic developments. Moreover, as DMD is purely a data-driven algorithm without the requirement for governing equations, it has been widely applied beyond fluid dynamics: in finance [109], video processing [110][111][112], epidemiology [113], robotics [114], and neuroscience [115]. As with many modal decomposition techniques, DMD is most often applied as a diagnostic to provide physical insight into a system.…”

Data-driven Spatiotemporal Modal Decomposition for Time Frequency Analysis

“…We will further assume that the state has been sampled evenly in time at some spacing ∆t at a total of n snapshots. DMD has become a popular tool to model dynamical systems in the fields of fluid mechanics, neuroscience, and image analysis [36,37,29,7].…”

Image-Based Model Predictive Control via Dynamic Mode Decomposition

“…DMD has seen many applications in areas such as video processing [17], fluid dynamics [31], and financial time series [21]. Many variants of DMD have been proposed to improve the robustness and…”