Anomaly Detection in Streaming Nonstationary Temporal Data

A Feature‐Based Procedure for Detecting Technical Outliers in Water‐Quality Data From In Situ Sensors

“…Our feature-based procedure has many advantages: (1) It can take the correlation structure of the water-quality variables into account when detecting outliers; (2) it can be applied to both univariate and multivariate problems; (3) the outlier scoring techniques that we consider are unsupervised, data-driven approaches and therefore do not require training data sets for the model building process and can be extended easily to other time series from other sites; (4) the outlier thresholds have a probabilistic interpretation as they are based on extreme value theory; (5) the approach has the ability to deal with irregular (unevenly spaced) time series; and (6) it can easily be extended to streaming data. In contrast to a batch scenario, which assumes that the entire data set is available prior to the analysis with the focus on detecting complete events, the streaming data scenario gives many additional challenges due to high velocity, unbounded, nonstationary data with incomplete events (Hill et al, 2009;Talagala, Hyndman, Smith-Miles, Kandanaarachchi, et al, 2019). In this paper, although our oddwater procedure is introduced as a batch method, it can easily be extended to streaming data such that it can provide near-real-time support using a sliding window technique.…”

“…This was the case in our study where no clear separation was visible between outliers and typical data points in the original data space, but a clear separation was obtained between the two sets of points once the one-sided derivative transformation was applied to the original series. Having this type of a separation between outliers and typical points is important before applying unsupervised outlier detection algorithms for high-dimensional data because the methods are usually based on the definition of outliers in terms of distance or density (Talagala, Hyndman, Smith-Miles, Kandanaarachchi, et al, 2019). Most of the outlier detection algorithms (KNN-SUM, KNN-AGG, NN-HD, COF, LOF, and INFLO) performed least well with the untransformed original series, demonstrating how data transformation methods can assist in improving the ability of outlier detection algorithms while maintaining low false detection rates.…”

“…In our modified algorithm, the NN-HD algorithm, we did not incorporate the clustering step of the HDoutliers algorithm because the data obtained from the two study sites are free from microclusters (Talagala, Hyndman, Smith-Miles, et al, 2019) and therefore free from the masking problem. Because the data sets have only local and global outliers, incorporating a clustering step that forms small clusters using a small ball with a fixed radius (the Leader Algorithm in Wilkinson, 2018) does not significantly change the structure of the data points in the high-dimensional data space.…”

Dimension Reduction for Outlier Detection Using DOBIN

“…To the best of our knowledge no formal dimension reduction techniques exist for outlier detection. It is common to use Principal Component Analysis (PCA) where dimension reduction is used when detecting outliers (e.g., Talagala et al, 2019;Hyndman, Wang & Laptev, 2015).…”

Constructing a Control Chart Using Functional Data

“…Briefly, the proposed methodology, which combines FDA and multivariate techniques, adapts the concept of the control chart based on a specific case of functional data and thereby presents a novel alternative for controlling facilities in which the data are obtained by continuous monitoring, as is the case with a great deal of process in the framework of Industry 4.0.Recently, diverse solutions have been provided that propose the modification and application of exponentially weighted moving average chart (EWMA) control charts [1]; control charts based on the fitting of autoregressive integrated moving average (ARIMA) models [2]; and the use of control charts for profiles, which is understood as the control of the parameters that define the relationship between two different CTQ variables [3][4][5][6]. Additionally, there are solutions that propose the use of techniques based on the control chart concept for anomaly detection such as machine learning techniques (neural networks and support vector machines, among others) and time series [7][8][9][10][11][12][13][14]. The increasingly common use of these tools is attributed to the fact that they adapt very well to the new paradigm of data (in the framework of Industry 4.0) defined by the continuous monitoring of numerous variables.…”