Searching and mining trillions of time series subsequences under dynamic time warping

Abstract

Most time series data mining algorithms use similarity search as a core subroutine, and thus the time taken for similarity search is the bottleneck for virtually all time series data mining algorithms. The difficulty of scaling search to large datasets largely explains why most academic work on time series data mining has plateaued at considering a few millions of time series objects, while much of industry and science sits on billions of time series objects waiting to be explored. In this work we show that by using a combination of four novel ideas we can search and mine truly massive time series for the first time. We demonstrate the following extremely unintuitive fact; in large datasets we can exactly search under DTW much more quickly than the current state-of-the-art Euclidean distance search algorithms. We demonstrate our work on the largest set of time series experiments ever attempted. In particular, the largest dataset we consider is larger than the combined size of all of the time series datasets considered in all data mining papers ever published. We show that our ideas allow us to solve higher-level time series data mining problem such as motif discovery and clustering at scales that would otherwise be untenable. In addition to mining massive datasets, we will show that our ideas also have implications for real-time monitoring of data streams, allowing us to handle much faster arrival rates and/or use cheaper and lower powered devices than are currently possible.

Flexible and Fast Similarity Search for Enriched Trajectories

“…Accordingly, many attempts have been made to reduce the computational cost of DTW [7], [14], [15], [19], [20]. Rath et al proposed the lower bound LB MV for the DTW distance of multivariate time series [17].…”

“…This is an extension of the lower bound LB Keogh [14] for the DTW distance of univariate time series. Some methods other than those using lower bounds have been proposed for accelerating the similarity search of time series data using DTW, such as early abandon [19] and data abstraction [20], which can be extended to the top-k problem of the enriched trajectories. However, all of these methods can be combined with techniques using lower bounds.…”

Less is More: Similarity of Time Series under Linear Transformations

“…In that respect, ED and DTW represent two extrema in the spectrum of temporal transformations: while Euclidean distance deals neither with dilation nor translation, dynamic time warping supports any possible warping path (and is assumed to be the best measure on average in [11]). We expected LTD to settle down somewhere half way between both measures.…”

“…The earlier we find a close pair, the more we benefit from lower bounds. In [11] various suggestions were made to find a good guess heuristically. We tested many datasets, but discuss only results on the 50words dataset.…”

A tighter lower bound estimate for dynamic time warping

“…Although DTW is a straightforward and intuitive sequence matching algorithm, its computational cost remains notoriously high, especially for matching in large corpora and mining trillions of time series [9]. Take DTW-based STD task for example.…”