An adaptive data cleaning framework: a case study of the water quality monitoring system in China

Chen, Zeng; Jiang, Peng; Xu, Hao; Li, Jun; Bychkov, Igor V.; Hmelnov, Alexei E.

doi:10.1080/02626667.2022.2060106

Cited by 4 publications

(3 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the method relies too much on the change of abnormal data over time, and the compensation effect of data with complex changes and continuous anomalies is poor [7]. Zeng Chen [8] and coauthors proposed the ADAPTIVE-EWT-MFE method, the ADAPTIVE-EWT-MFE method uses EWT and MFE to clean water quality data. It decomposes the data into IMFs and introduces an adaptive threshold parameter based on MFE to filter high-frequency noise without distorting noise-free data.…”

Section: Introductionmentioning

confidence: 99%

A Big Data Cleaning Method for Drinking-Water Streaming Data

Gai

Zhang

Thanh

2023

Braz. arch. biol. technol.

View full text Add to dashboard Cite

A HA_Cart_AdaBoost model is proposed to clean the data in drinking-water-quality data. First, the data that do not follow the normal distribution are regarded as outliers and eliminated. Next, the optimal control theory of nonlinear partial differential equations (PDEs) is introduced into the cart decision tree, and the cart decision with the specified depth is used. As a weak classifier of AdaBoost, the tree uses the HA_Cart_AdaBoost model to compensate for the eliminated data, then it fits and predicts the missing values of the data stream, realizes the cleaning of drinking-water-quality data, and finally uses the big data Hadoop architecture for real-time storage and analysing streaming data. The experimental results show that compared with the most advanced data cleaning methods, after the optimal control theory of nonlinear PDEs is introduced into the cart decision tree, the stability and accuracy of the HA_Cart_AdaBoost model for water quality data cleaning are greatly improved. Taking pH as an example, the HA_Cart_AdaBoost model shows a minimum improvement of 2.25% and a maximum improvement of 53.33% in terms of RMSE, and a minimum improvement of 13.51% and a maximum improvement of 78.08% in terms of MAE.

show abstract

Section: Introductionmentioning

confidence: 99%

A Big Data Cleaning Method for Drinking-Water Streaming Data

Gai

Zhang

Thanh

2023

Braz. arch. biol. technol.

View full text Add to dashboard Cite

show abstract

“…Wu Jianhua studied the water quality and pollution of groundwater, and explored the main influencing factors of groundwater pollution after analyzing the water quality parameters [7]. Chen Zeng studied the noise removal of water quality data and proposed an adaptive noise removal method, which provided a reference for the relevant research on noise removal of water quality data [8]. Zhu Weiyu developed a method for surface water quality detection based on regression model, which could effectively detect water quality anomalies [9].…”

Section: Introductionmentioning

confidence: 99%

Construction of Water Pollution Early Warning System Based on Clustering Algorithm and Machine Learning

2021

WPPCP

View full text Add to dashboard Cite

With the rapid development of industry and towns, environmental pollution accidents occur frequently. It is of great significance to establish a sound environmental monitoring and early warning system to ensure the safety of water environment and people's water use. On this basis, this paper deeply discussed the early warning of water pollution, and proposed a water pollution early warning system based on clustering algorithm. The research showed that when the water pollution early warning system was used to monitor and warn the harmful substances such as formaldehyde, cyanide and Chemical Oxygen Demand (COD), the average absolute error of the water pollution early warning system when monitoring the four harmful substances was about 0.67, and the mean square error of the water pollution early warning system when monitoring the four harmful substances was about 0.66. From the perspective of the mean absolute error and mean square error, it could be seen that the water pollution early warning system based on clustering algorithm had good detection accuracy. In addition, in the evaluation study of the water pollution early warning system, this paper reached the conclusion that most of the monitoring and early warning technicians were relatively approved of the water pollution early warning system.

show abstract

“…Chen Zeng's robust detection of patterns under low signal-to-noise ratio was a basic challenge to analyze high-frequency data, especially in water quality monitoring. He proposed an adaptive method based on empirical wavelet transform and multi-scale fuzzy entropy to achieve time series data cleaning [8]. Meng Qingxuan proposed a data cleaning method based on improved balanced iterative reduction and hierarchical clustering algorithm.…”

Section: Introductionmentioning

confidence: 99%

Time Series Data Cleaning and Early Warning of River Basin Water Quality

2020

WPPCP

View full text Add to dashboard Cite

In order to effectively control the safety of water environment and promote the continuous improvement of water environment quality, all national environmental monitoring stations in the basin should conduct intensive monitoring of water quality, so that the environmental protection department can timely understand the water quality dynamics and make decisions on early warning and water quality analysis. How to monitor and warn watershed water quality data has become one of the current research hotspots. However, most water quality sensors often suffer from daily maintenance difficulties, database input errors and sensor measurement errors. Therefore, this paper analyzed the framework and process of data cleaning of time series, then optimized and analyzed the early warning model, and finally analyzed the early warning effect through abnormal early warning algorithm. The water quality warning effect after data cleaning was 15.6% higher than that before data cleaning, and the indicator detection effect was 11.2% higher than that before data cleaning. In short, data cleaning of time series and early warning model can improve water quality.

show abstract

An adaptive data cleaning framework: a case study of the water quality monitoring system in China

Cited by 4 publications

References 28 publications

A Big Data Cleaning Method for Drinking-Water Streaming Data

A Big Data Cleaning Method for Drinking-Water Streaming Data

Construction of Water Pollution Early Warning System Based on Clustering Algorithm and Machine Learning

Time Series Data Cleaning and Early Warning of River Basin Water Quality

Contact Info

Product

Resources

About