Streaming changepoint detection for transition matrices

Abstract

Sequentially detecting multiple changepoints in a data stream is a challenging task. Difficulties relate to both computational and statistical aspects, and in the latter, specifying control parameters is a particular problem. Choosing control parameters typically relies on unrealistic assumptions, such as the distributions generating the data, and their parameters, being known. This is implausible in the streaming paradigm, where several changepoints will exist. Further, current literature is mostly concerned with streams of continuous-valued observations, and focuses on detecting a single changepoint. There is a dearth of literature dedicated to detecting multiple changepoints in transition matrices, which arise from a sequence of discrete states. This paper makes the following contributions: a complete framework is developed for adaptively and sequentially estimating a Markov transition matrix in the streaming data setting. A change detection method is then developed, using a novel moment matching technique, which can effectively monitor for multiple changepoints in a transition matrix. This adaptive detection and estimation procedure for transition matrices, referred to as ADEPT-M, is compared to several change detectors on synthetic data streams, and is implemented on two real-world data streams – one consisting of over nine million HTTP web requests, and the other being a well-studied electricity market data set.

Cluster-based stability evaluation in time series data sets

“…When regarding works dealing with outlier detection in time series, various definitions of the term outlier can be found. Many approaches consider only single conspicuous data points such as additive outliers or change points [24,37,43] and focus on a single time series [1,39]. However, in our context the detection of anomalous subsequences is considered, so that only algorithms, which either handle outlier subsequences or analyse the group behavior of multiple time series over time, are relevant.…”

“…Both results are very similar to each other, as they differ only at one timestamp and that is the last one. Each method detects all three outlier sequences (42,43,44) in the first four timestamps. At time 5, all approaches are in agreement that there are only two outliers: 42 and 43.…”