Estimation, Forecasting, and Anomaly Detection for Nonstationary Streams Using Adaptive Estimation

Abstract

Streaming data provides substantial challenges for data analysis. From a computational standpoint, these challenges arise from constraints related to computer memory and processing speed. Statistically, the challenges relate to constructing procedures that can handle so-called concept drift -the tendency of future data to have different underlying properties to current and historic data. The issue of handling structure, such as trend and periodicity, remains a difficult problem for streaming estimation. We propose RAC (Real-Time Adaptive Component), a penalized-regression modelling framework which satisfies the computational constraints of streaming data, and provides capability for dealing with concept drift. At the core of the estimation process are techniques from adaptive filtering. The RAC procedure adopts a specified basis to handle local structure, along with a LASSO-like penalty procedure to handle over-fitting. We enhance the RAC estimation procedure with a streaming anomaly detection capability. Experiments with simulated data suggest the procedure can be considered as a competitive tool for a variety of scenarios, and an illustration with real cyber-security data further demonstrates the promise of the method.

Detecting unexpected growths in health technologies expenditures: the case of MIPRES in Colombia

“…The models used were Seasonal Hybrid Extreme Studentized Deviate (S-H-ESD) and Two-Stage Dataset Shift-detection based on an Exponentially Weighted Moving Average (TSSD-EWMA). These techniques are commonly employed and known for their robustness in handling non-stationary time series [ 14 – 19 ]. The final set of anomalies consists of the series in which all three observations in 2022Q1 are detected as outliers by both algorithms.…”

Online Attention Enhanced Differential and Decomposed LSTM for Time Series Prediction

“…W ITH the advent of big data era, time series data widely exists in the fields of traffic monitoring [1], [2], electrical system [3], [4], meteorological and environmental measurement [5]- [7], financial services [8], [9], bioinformatics [10]- [12], image processing [13]- [16], etc., and has gradually become an important part of big data. In some applications, such as proactive resource scheduling in the stream processing platforms [17], [18] and exchange management in finance, time series prediction is the premise and key to correct decision-making [19], [20], thus getting more and more attention. However, time series continues to grow over time, exhibiting characteristics such as time-varying and sudden changes [21], making it difficult to make accurate predictions, and the prediction lags severely (i.e.…”

Detecting Outliers Using Modified Recursive PCA Algorithm For Dynamic Streaming Data

“…Later, Schifano et al [25] and Wang et al [29] detected outliers with standardized predictive residuals and to test for outliers in the n-th data where after estimating the parameters with the Bayesian framework method. Hoeltgebaum et al [17] identified anomalies using The Hall-Buckley Eggleson (HBE) method after estimating parameters using the LASSO method. Then, Ippel et al [19] estimated the parameters recursively with stochastic gradient descent.…”