Active learning for anomaly detection in environmental data

Russo, Stefania; Lürig, Moritz; Hao, Wenjin; Matthews, Blake; Villez, Kris

doi:10.1016/j.envsoft.2020.104869

Cited by 41 publications

(22 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the time series of ecosystem dynamics, we first performed an outlier analysis by excluding values higher than three times the median absolute deviation of all values in a sliding window (Leys et al 2013) of one day window size (15-minute interval = 96 data points). In addition to outlier removal, we visually inspected the data and manually removed anomalous periods from the data (<2%; for more details refer to Russo et al [2020]). After aggregating four measurement points to one per hour (from 96 to 24 data points per day), we calculated mean and coefficient of variation (hereafter CV) of the aggregated data within windows that were sized one week (7 × 24 = 168 data points).…”

Section: Discussionmentioning

confidence: 99%

Non‐additive effects of foundation species determine the response of aquatic ecosystems to nutrient perturbation

Lürig

Narwani

Penson

et al. 2021

Ecology

Self Cite

View full text Add to dashboard Cite

Eutrophication is a persistent threat to aquatic ecosystems worldwide. Foundation species, namely those that play a central role in the structuring of communities and functioning of ecosystems, are likely important for the resilience of aquatic ecosystems in the face of disturbance. However, little is known about how interactions among such species influence ecosystem responses to nutrient perturbation. Here, using an array (N = 20) of outdoor experimental pond ecosystems (15,000 L), we manipulated the presence of two foundation species, the macrophyte Myriophyllum spicatum and the mussel Dreissena polymorpha, and quantified ecosystem responses to multiple nutrient disturbances, spread over two years. In the first year, we added five nutrient pulses, ramping up from 10 to 50 μg P/L over a 10‐week period from mid‐July to mid‐October, and in the second year, we added a single large pulse of 50 μg P/L in mid‐October. We used automated sondes to measure multiple ecosystems properties at high frequency (15‐minute intervals), including phytoplankton and dissolved organic matter fluorescence, and to model whole‐ecosystem metabolism. Overall, both foundation species strongly affected the ecosystem responses to nutrient perturbation, and, as expected, initially suppressed the increase in phytoplankton abundance following nutrient additions. However, when both species were present, phytoplankton biomass increased substantially relative to other treatment combinations: non‐additivity was evident for multiple ecosystem metrics following the nutrient perturbations in both years but was diminished in the intervening months between our perturbations. Overall, these results demonstrate how interactions between foundation species can cause surprisingly strong deviations from the expected responses of aquatic ecosystems to perturbations such as nutrient additions.

show abstract

Section: Discussionmentioning

confidence: 99%

Non‐additive effects of foundation species determine the response of aquatic ecosystems to nutrient perturbation

Lürig

Narwani

Penson

et al. 2021

Ecology

Self Cite

View full text Add to dashboard Cite

show abstract

“…In assessing algorithm performance, we defer to technician labels as a benchmark. However, the quality control process is subjective (Jones et al, 2018) and data are not perfectly labeled, making reliance on technician labels as a gold standard problematic (Russo et al, 2020). In the LRO data, we identified numerous cases where it was unclear why some data points were labeled and others were not (see Appendix C), which may be due to multiple technicians and evolving protocols, among other reasons.…”

Section: Anomaly Detection Examplementioning

confidence: 99%

“…Another regression technique based on a previous sequence of data is Long Short-Term Memory (LSTM), a class of Artificial Neural Networks (ANNs). Though applications to environmental data anomalies to date are limited, LSTM models have been used to reconstruct time series to detect anomalies in other fields (Hundman et al, 2018;Lindemann et al, 2019;Malhotra et al, 2016;Yin et al, 2020), and other ANN model types have been used for environmental anomaly detection (Hill and Minsker, 2010;Russo et al, 2020). Other algorithms that show promise for time series regression include Prophet, a time series forecasting method developed by Facebook with focus on business applications (Taylor and Letham, 2018), and Hierarchical Temporal Memory (HTM) (Ahmad et al, 2017).…”

Section: A4 Regression Approachesmentioning

confidence: 99%

“…Feature based methods comprise another class of anomaly detectors commonly used for discrete data (Tan et al, 2019), which some authors have applied to environmental time series (Leigh et al, 2018;Russo et al, 2020;Talagala et al, 2019). Unlike regression methods, feature based methods do not make a prediction of the observation.…”

Section: A5 Feature Based Approachesmentioning

confidence: 99%

“…As a result, there is a recognized need for automating and improving quality control post processing for high frequency in situ sensor data. In this vein, automated, data driven techniques to detect anomalies in streaming sensor data are documented in the realm of research (Hill and Minsker, 2010;Leigh et al, 2018;Russo et al, 2020;Talagala et al, 2019); however, they are unfamiliar to practitioners, generally lack robust and accessible software implementations, and are not typically reproducible. Furthermore, while basic checks and more complex algorithms may identify and flag potentially erroneous values (e.g., Dereszynski and Dietterich, 2007;Hill et al, 2009;Taylor and Loescher, 2013), these procedures are generally not capable of applying corrective actions.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Toward automating post processing of aquatic sensor data

Jones¹,

Jones²,

Horsburgh³

2022

Preprint

View full text Add to dashboard Cite

Sensors measuring environmental phenomena at high frequency commonly report anomalies related to fouling, sensor drift and calibration, and datalogging and transmission issues. Suitability of data for analyses and decision making often depends on manual review and adjustment of data. Machine learning techniques have potential to automate identification and correction of anomalies, streamlining the quality control process. We explored approaches for automating anomaly detection and correction of aquatic sensor data for implementation in a Python package (PyHydroQC). We applied both classical and deep learning time series regression models that estimate values, identify anomalies based on dynamic thresholds, and offer correction estimates. Techniques were developed and performance assessed using data reviewed, corrected, and labeled by technicians in an aquatic monitoring use case. Auto-Regressive Integrated Moving Average (ARIMA) consistently performed best, and aggregating results from multiple models improved detection. PyHydroQC includes custom functions and a workflow for anomaly detection and correction.

show abstract

Can machine learning accelerate process understanding and decision‐relevant predictions of river water quality?

Varadharajan

Appling

Arora

et al. 2022

Hydrological Processes

View full text Add to dashboard Cite

The global decline of water quality in rivers and streams has resulted in a pressing need to design new watershed management strategies. Water quality can be affected by multiple stressors including population growth, land use change, global warming, and extreme events, with repercussions on human and ecosystem health. A scientific understanding of factors affecting riverine water quality and predictions at local to regional scales, and at sub‐daily to decadal timescales are needed for optimal management of watersheds and river basins. Here, we discuss how machine learning (ML) can enable development of more accurate, computationally tractable, and scalable models for analysis and predictions of river water quality. We review relevant state‐of‐the art applications of ML for water quality models and discuss opportunities to improve the use of ML with emerging computational and mathematical methods for model selection, hyperparameter optimization, incorporating process knowledge into ML models, improving explainablity, uncertainty quantification, and model‐data integration. We then present considerations for using ML to address water quality problems given their scale and complexity, available data and computational resources, and stakeholder needs. When combined with decades of process understanding, interdisciplinary advances in knowledge‐guided ML, information theory, data integration, and analytics can help address fundamental science questions and enable decision‐relevant predictions of riverine water quality.

show abstract

Active learning for anomaly detection in environmental data

Cited by 41 publications

References 35 publications

Non‐additive effects of foundation species determine the response of aquatic ecosystems to nutrient perturbation

Non‐additive effects of foundation species determine the response of aquatic ecosystems to nutrient perturbation

Toward automating post processing of aquatic sensor data

Can machine learning accelerate process understanding and decision‐relevant predictions of river water quality?

Contact Info

Product

Resources

About