Anomaly Detection Using a Sliding Window Technique and Data Imputation with Machine Learning for Hydrological Time Series

Abstract

Water level data obtained from telemetry stations typically contains large number of outliers. Anomaly detection and a data imputation are necessary steps in a data monitoring system. Anomaly data can be detected if its values lie outside of a normal pattern distribution. We developed a median-based statistical outlier detection approach using a sliding window technique. In order to fill anomalies, various interpolation techniques were considered. Our proposed framework exhibited promising results after evaluating with F1-score and root mean square error (RMSE) based on our artificially induced data points. The present system can also be easily applied to various patterns of hydrological time series with diverse choices of internal methods and fine-tuned parameters. Specifically, the Spline interpolation method yielded a superior performance on non-cyclical data while the long short-term memory (LSTM) outperformed other interpolation methods on a distinct tidal data pattern.

A Quantitative Modeling and Prediction Method for Sustained Rainfall-PM2.5 Removal Modes on a Micro-Temporal Scale

“…Firstly, according to the definitions of the SRS and RES proposed in Section 3.1, a small-scale sliding window is applied to process the hourly rainfall time series R t , hourly PM 2.5 time series C t and hourly meteorological observations, setting up to capture all of the sustained rainfall processes in the historical data [40].…”

An intelligent intrusion detection system

“…To create and organize them, are described two approaches [67]: Sequential and Sliding. In sequential observation windows ( also known as non-overlapping sliding windows ) [68], as the name suggests, the observation windows have the same size and are organized In the sliding approach ( also known as overlapping sliding windows ) [69], a numerical value known as sliding value is defined, which indicates how many sampling windows the start of current observation window is deviated from the previous. An example is illustrated in The sliding windows generates more observation windows than the sequential approach and, consequently, more samples of the user's activity, while maintaining the same window size.…”

Hydrological Time Series Clustering: A Case Study of Telemetry Stations in Thailand

“…Some of these data tend to behave similarly, and some behave completely different from others. Clustering these data into distinct groups is an initial step prior to further analysis in water management analytics, which includes anomaly detection and data imputation, as well as a forecasting model [7]. Due to diverse behaviors of water level data across stations, the developed models for these analytics tasks could underperform significantly.…”