Generic and Scalable Framework for Automated Time-series Anomaly Detection

Anomaly Detection As-a-Service

“…However, many anomaly detection techniques simply process data in batches and for this reason are unsuitable for real-time streaming applications. This is the case of Symbolic Aggregate Approximation [18], which has been used to find the most unusual subsequences within a time series, the supervised learning approach by Hermine et al [3], which leverages a combination of a Bayesian maximum likelihood classifier and a linear regression model to spot anomalies in temporal structures, and the Netflix's robust principle component analysis (RPCA) method [22], and Yahoo's EGADS [19].…”

Adaptive Anomaly Detection in Performance Metric Streams

“…This approach is generally not application or infrastructure agnostic since they often require intrusive source instrumentation [4] and hence hard to generalize for diverse services and use-cases [8], such as in the cloud computing scenario. This has led to renewed interest within the academic and industrial community in performance anomaly detection techniques that are both scalable and versatile [1], [2], [8]- [13]. However, advancement in service deployment, complex service workloads, and dynamism introduced in cloud computing environments, have led to interesting contextual behaviour in performance metric data that are hardly explored by existing systems.…”

“…However, in performance diagnosis, many techniques rely on the premise that statistical distribution of metric measurements is either known or static [12], [20], that monitoring data is available a priori for model selection and training [1], and that what constitute normal and/or abnormal behaviour is well delineated [4]. This somewhat limits their applicability in realtime scenarios where they may generate many false-alarms as baselines become obsolete [8]. This is not desirable since the cost of missing an anomaly is often very high.…”

“…Such points are considered 'unusual' for reasons such as exhibiting lack of continuity with immediate observations or deviation from the stream's short-or long-term behaviour [25]. In general, having a precognition of the form of M t can provide clues on assumptions and analysis approach [8]. However, since it is non-trivial to estimate this accurately in online settings, methods that are somewhat agnostic to temporal patterns of the stream are desirable.…”

“…For example, point maxima thresholds and 3-sigma rules have been used in [20] to detect latency SLA violations, in EGADS [8] to detect change-points given expected behaviour of a metric stream, in CloudPD [12] to detect latency degradation, in E2EProf [46] to detect spikes in request latencies, and in CPI2 [11] to detect anomalies in per job cycle-per-instruction (CPI) of Google's web-search cluster. In addition, [9] assumes Gaussian distribution to define adaptive thresholds for uncovering anomalous behaviour in performance of an advertising service.…”