Reconstructing Missing and Anomalous Data Collected from High-Frequency In-Situ Sensors in Fresh Waters

Abstract

In situ sensors that collect high-frequency data are used increasingly to monitor aquatic environments. These sensors are prone to technical errors, resulting in unrecorded observations and/or anomalous values that are subsequently removed and create gaps in time series data. We present a framework based on generalized additive and auto-regressive models to recover these missing data. To mimic sporadically missing (i) single observations and (ii) periods of contiguous observations, we randomly removed (i) point data and (ii) day- and week-long sequences of data from a two-year time series of nitrate concentration data collected from Arikaree River, USA, where synoptically collected water temperature, turbidity, conductance, elevation, and dissolved oxygen data were available. In 72% of cases with missing point data, predicted values were within the sensor precision interval of the original value, although predictive ability declined when sequences of missing data occurred. Precision also depended on the availability of other water quality covariates. When covariates were available, even a sudden, event-based peak in nitrate concentration was reconstructed well. By providing a promising method for accurate prediction of missing data, the utility and confidence in summary statistics and statistical trends will increase, thereby assisting the effective monitoring and management of fresh waters and other at-risk ecosystems.

Understanding links between water-quality variables and nitrate concentration in freshwater streams using high frequency sensor data

“…The findings of this study are highly relevant for scientists and managers responsible for insitu monitoring in rivers. As mentioned above, gaps in in-situ sensor data are common and the methods demonstrated here could be applied to the problem of missing data imputation [50] and provide a more holistic description of nitrate dynamics. This is particularly important when financial resources are limited and decisions must be made about which sensors to buy and which water-quality variables to measure.…”

CCMMI: A class center-based multiple mixed imputation method

“…Missing data refers to the incomplete sample records in the dataset, which may be missing in one or more variables of some samples. The phenomenon of missing data in the real world exists in many elds, such as industry 7 , medicine 8,9 , business 10 and scienti c research 11 . There are various reasons for data loss, mainly divided into mechanical factors and human factors.…”

Intelligent Identification and Order-Sensitive Correction Method of Outliers from Multi-Data Source Based on Historical Data Mining

“…The abnormal data are distinguished via multi-sensor collaboration, and the stack de-noising automatic encoder was used to realize data cleaning. The literature [10] proposed a multi-source missing data and abnormal data correction framework based on the generalized addition and auto-regression model, to solve the problem of multi-source data missing and abnormal data processing in aquatic environmental monitoring, which effectively assisted the monitoring and management of freshwater eco systems. According to the spatial distribution characteristics of abnormal and normal data pixels in wind power curve images, the literature [11] extracted abnormal and normal data pixels through image processing to achieve rapid cleaning of abnormal data of wind turbines.…”