Toward automating post processing of aquatic sensor data

Abstract

Sensors measuring environmental phenomena at high frequency commonly report anomalies related to fouling, sensor drift and calibration, and datalogging and transmission issues. Suitability of data for analyses and decision making often depends on manual review and adjustment of data. Machine learning techniques have potential to automate identification and correction of anomalies, streamlining the quality control process. We explored approaches for automating anomaly detection and correction of aquatic sensor data for implementation in a Python package (PyHydroQC). We applied both classical and deep learning time series regression models that estimate values, identify anomalies based on dynamic thresholds, and offer correction estimates. Techniques were developed and performance assessed using data reviewed, corrected, and labeled by technicians in an aquatic monitoring use case. Auto-Regressive Integrated Moving Average (ARIMA) consistently performed best, and aggregating results from multiple models improved detection. PyHydroQC includes custom functions and a workflow for anomaly detection and correction.

Reconstructing Missing and Anomalous Data Collected from High-Frequency In-Situ Sensors in Fresh Waters

“…Applying the same R 2 coefficient to our simulations, we achieved R 2 values between 0.976 and 0.997 when 40% of the dataset was removed and between 0.119 and 0.994 (first quartile = 0.84) when sequences of 10 days were removed, indicating that our method attains a comparatively good performance, particularly for point and short periods of missing data. Blending ARIMA forecasts and backcasts has also shown promise for reconstruction of sensor-based water quality data, including temperature, pH, specific conductance, and dissolved oxygen [ 58 ]. However, we are unable to compare our results with this work given that performance of the correction method was assessed by comparing the ARIMA-based results with corrections done manually by technicians.…”

Value and limitations of Machine Learning in high-frequency nutrient data for gap- filling, forecasting, and transport process interpretation

“…It is often the case that sensors and autoanalyzers have technical problems which can result in a signi cant number of gaps in the data. The post processing of the data, including identi cation of errors and lling missing values can be time consuming and the nal result depends on the individual who does the post processing (Jones et al, 2021). Furthermore, the amount of data collected by high-frequency sensors can easily exceed the amount of data that can be treated manually (Dupas et al, 2015;Kirchner & Neal, 2013).…”

Improving the management of drained agricultural areas to optimize the retention of iron-associated phosphorus

“…Moreover, standardization of post-processing high-frequency data is also challenging as currently, the final result depends largely on the person who did the data analysis (Jones et al, 2021). To observe the bigger picture in nutrient transport and to evaluate trends related to rain variability and climate change, more long-term high-quality continuous water quality data series are needed.…”

“…It is often the case that sensors and autoanalyzers have technical problems which can result in a significant number of gaps in the data. The post processing of the data, including identification of errors and filling missing values can be time consuming and the final result depends on the individual who does the post processing (Jones et al, 2021). Furthermore, the amount of data collected by high-frequency sensors can easily exceed the amount of data that can be treated manually (Dupas et al, 2015;Kirchner & Neal, 2013).…”