On the nature and types of anomalies: a review of deviations in data

Foorthuis, Ralph

doi:10.1007/s41060-021-00265-1

Cited by 70 publications

(22 citation statements)

References 264 publications

(506 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Decades ago, robust statistics based on median values were developed for minimizing the impact of aberrant outliers in the data (i.e., assuming the worst case scenario), the cause of which is beyond the experience or knowledge of the data analyst. However, today those aberrant outliers can now be tracked and ruled out by quality control and database management methods (Schultz et al, 2017), and therefore, the problem of aberrant outliers is hardly an issue any more (but the identification of possible anomalies is still one of the most challenging problems for the research community, Foorthuis, 2021). Under the circumstance that the aberrant outliers are removed and the data record is sufficiently long, most techniques can describe the central tendency properly and give similar trend estimators (either mean-or median-based estimator), but this also implies these estimations cannot be used to represent the change of the extreme events.…”

Section: Discussionmentioning

confidence: 99%

Trend detection of atmospheric time series

Chang

Schultz

et al. 2021

Elementa: Science of the Anthropocene

View full text Add to dashboard Cite

This paper is aimed at atmospheric scientists without formal training in statistical theory. Its goal is to (1) provide a critical review of the rationale for trend analysis of the time series typically encountered in the field of atmospheric chemistry, (2) describe a range of trend-detection methods, and (3) demonstrate effective means of conveying the results to a general audience. Trend detections in atmospheric chemical composition data are often challenged by a variety of sources of uncertainty, which often behave differently to other environmental phenomena such as temperature, precipitation rate, or stream flow, and may require specific methods depending on the science questions to be addressed. Some sources of uncertainty can be explicitly included in the model specification, such as autocorrelation and seasonality, but some inherent uncertainties are difficult to quantify, such as data heterogeneity and measurement uncertainty due to the combined effect of short and long term natural variability, instrumental stability, and aggregation of data from sparse sampling frequency. Failure to account for these uncertainties might result in an inappropriate inference of the trends and their estimation errors. On the other hand, the variation in extreme events might be interesting for different scientific questions, for example, the frequency of extremely high surface ozone events and their relevance to human health. In this study we aim to (1) review trend detection methods for addressing different levels of data complexity in different chemical species, (2) demonstrate that the incorporation of scientifically interpretable covariates can outperform pure numerical curve fitting techniques in terms of uncertainty reduction and improved predictability, (3) illustrate the study of trends based on extreme quantiles that can provide insight beyond standard mean or median based trend estimates, and (4) present an advanced method of quantifying regional trends based on the inter-site correlations of multisite data. All demonstrations are based on time series of observed trace gases relevant to atmospheric chemistry, but the methods can be applied to other environmental data sets.

show abstract

Section: Discussionmentioning

confidence: 99%

Trend detection of atmospheric time series

Chang

Schultz

et al. 2021

Elementa: Science of the Anthropocene

View full text Add to dashboard Cite

show abstract

“…There are various types and subtypes of anomalies, with 3 broad, 9 basic and 63 subtypes, which have been previously studied. These should be understood in the analysis of healthcare associated data as further research work is done in anomaly detection [ 68 ].…”

Section: Basic Categorization Of Anomaly Detectionmentioning

confidence: 99%

Anomaly Detection Framework for Wearables Data: A Perspective Review on Data Concepts, Data Analysis Algorithms and Prospects

Sunny¹,

Patro²,

Karnani³

et al. 2022

Sensors

View full text Add to dashboard Cite

Wearable devices use sensors to evaluate physiological parameters, such as the heart rate, pulse rate, number of steps taken, body fat and diet. The continuous monitoring of physiological parameters offers a potential solution to assess personal healthcare. Identifying outliers or anomalies in heart rates and other features can help identify patterns that can play a significant role in understanding the underlying cause of disease states. Since anomalies are present within the vast amount of data generated by wearable device sensors, identifying anomalies requires accurate automated techniques. Given the clinical significance of anomalies and their impact on diagnosis and treatment, a wide range of detection methods have been proposed to detect anomalies. Much of what is reported herein is based on previously published literature. Clinical studies employing wearable devices are also increasing. In this article, we review the nature of the wearables-associated data and the downstream processing methods for detecting anomalies. In addition, we also review supervised and un-supervised techniques as well as semi-supervised methods that overcome the challenges of missing and un-annotated healthcare data.

show abstract

“…Anomaly type (1) Energy "temporary change (ST-VIIb)" and "variation change (ST-VIIf)" [11], "CONSTANT fault" [38] Power "temporary change (ST-VIIb)" and "variation change (ST-VIIf)" [11] Anomaly type (2) Energy "temporary change (ST-VIIb)" and "variation change (ST-VIIf)" [11], "stuck-at fault" [29], "stuck fault" [48] Power "temporary change (ST-VIIb)" and "variation change (ST-VIIf)" [11] Anomaly type (3) Energy "level shift (ST-VIIc)" [11] Power "local additive (ST-IVe)" [11], fault" [29], "SHORT fault" [38], "spike fault" [48] Anomaly type (4) Energy "level shift (ST-VIIc)" [11] Power "local additive (ST-IVe)" [11], "outlier fault" [29], "SHORT fault" [38], "spike fault" [48] C PARAMETERS…”

Section: Time Series Matching Classes In Literaturementioning

confidence: 99%

Modeling and generating synthetic anomalies for energy and power time series

Turowski

Weber

Neumann

et al. 2022

Proceedings of the Thirteenth ACM International Conference on Future Energy Systems

View full text Add to dashboard Cite

With the development of the smart grid, the number of recorded energy and power times series increases noticeably. This increase allows for the automation of smart grid applications such as load forecasting and load management. This automation, however, requires data that only represents the typical behavior of the system. To ensure that such data is available, detecting the anomalies often present in recorded data is important. As a result, anomaly detection methods are a recent research topic. However, their development is often limited by undefined anomaly characteristics and a lack of labeled anomalous data. To overcome this challenge, we propose a method that generates synthetic anomalies based on real-world anomalies that can be inserted into energy and power time series. For this, we analyze real energy and power time series to identify four types of commonly occurring anomalies. Given the identified anomaly types, we formally model each type and use these models to insert synthetic anomalies of each type into arbitrary energy or power time series. We show that our method is not only capable of generating synthetic anomalies with real-world properties, but also beneficial for training supervised anomaly detection methods. CCS CONCEPTS• Computing methodologies → Anomaly detection.

show abstract

On the nature and types of anomalies: a review of deviations in data

Cited by 70 publications

References 264 publications

Trend detection of atmospheric time series

Trend detection of atmospheric time series

Anomaly Detection Framework for Wearables Data: A Perspective Review on Data Concepts, Data Analysis Algorithms and Prospects

Modeling and generating synthetic anomalies for energy and power time series

Contact Info

Product

Resources

About