A method for automatic validation of long time series of data in urban hydrology

Abstract

Modelling in urban hydrology is largely based on the analysis of long time series of data. The quality of the results strongly depends on the quality of the data used. Doubtful or wrong data must be detected and eventually substituted by reliable ones when it is feasible before any further exploitation. This paper deals with the development of an automatic pre-validation procedure that detects doubtful and not reliable data, in order to facilitate their interpretation. This procedure consists in applying a set of seven tests based on the following criteria: the functioning state of the sensor, the physical range of the quantity, the locally realistic range, the duration since the last maintenance of the sensor, the signal's gradient, material redundancy and analytical redundancy. The results of the tests are coded with the letter A for reliable values, B for doubtful values and C for wrong values. After this automatic prevalidation, the ultimate validation of values marked B and C is carried out manually by the operator, with the assistance of specifically developed visual and graphical tools.

Evaluation of the Performance and the Predictive Capacity of Build-Up and Wash-Off Models on Different Temporal Scales

“…Raw measurements obtained directly from site monitoring cannot be used before undergoing a validation procedure in order to eliminate non reliable measurements and interpolating when possible missing data points [28,29]. Automatic pre-validation is first developed to highlight wrong and doubtful data by associating a mark that reflects the validity of each measurement of turbidity, followed by a final manual validation.…”

“…Automatic pre-validation is first developed to highlight wrong and doubtful data by associating a mark that reflects the validity of each measurement of turbidity, followed by a final manual validation. The technique is inspired by the work of Mourad and Bertrand-Krajewski [29] and the software EVOHE [30] but modified to fit this study case since turbidity measurements are obtained using one turbidimeter at the inlet of the drainage network.…”

Enhanced Monitoring of Environmental Processes

“…The objective of the GLR-based PCA fault detection technique is to detect the additive fault,  , with the maximum detection probability for a given false alarm. Here, the fault detection task can be considered as a hypothesis testing problem with consideration of two possible hypotheses: null hypothesis of no change 0 H , where measurements vector X , is fault-free, and the change-point alternative hypothesis 1 H , where X contains a fault, and thus X is no longer categorized by the fault-free PCA model (4 (18) It is assumed that the residual in Equation (17) The algorithm which studies the developed GLR-based PCA fault detection technique is presented in Algorithm 1. The GLR-based PCA is proposed to detect the faults in the residual vector obtained from the PCA model, through which the GLR test is used for each residual vector, R .…”

“…These techniques are useful since operation safety and the high quality products are some of the core objectives in the industry applications. Faults detection has been performed manually using data visualization tools [1], but these tools are time consuming for real-time detection in streaming data. Recently, researchers have proposed automated statistical and machine learning methods, such as: nearest neighbor [2], clustering [3], Marie-France Destain is with the Department of Biosystems Engineering, University of Liege, Belgium (e-mail: mfdestain@ulg.ac.be).…”

Data acquisition, validation and forecasting for a combined sewer network

“…Techniques for the rehabilitation of missing data have been developed (Bennis et al [2]). Little research has been done as far as the automated review (pre-validation) of environmental data is concerned (Mourad and Bertrand-Krajewski [8]). …”