Metamorphic Testing of Machine Learning and Conceptual Hydrologic Models

Abstract

Abstract. Predicting the response of hydrologic systems to modified driving forces, beyond patterns that have occurred in the past, is of high importance for estimating climate change impacts or the effect of management measures. This kind of predictions requires a model, but the impossibility of testing such predictions against observed data makes it still difficult to estimate their reliability. Metamorphic testing offers a methodology for assessing models beyond validation with real data. It consists of defining input changes for which the expected responses are assumed to be known at least qualitatively, and to test model behavior for consistency with these expectations. To increase the gain of information and reduce the subjectivity of this approach, we extend this methodology to a multi-model approach and include a sensitivity analysis of the predictions to training or calibration options. This allows us to quantitatively analyse differences in predictions between different model structures and calibration options in addition to the qualitative test to the expectations. In our case study, we apply this approach to selected conceptual and machine learning hydrological models calibrated to basins from the CAMELS data set. Our results confirm the superiority of the machine learning models over the conceptual hydrologic models regarding the quality of fit during calibration and validation periods. However, we also find that the response of machine learning models to modified inputs can deviate from the expectations and the magnitude and even the sign of the response can depend on the training data. In addition, even in cases in which all models passed the metamorphic test, there are cases in which the quantitative response is different for different model structures. This demonstrates the importance of this kind of testing beyond the usual calibration-validation analysis to identify potential problems and stimulate the development of improved models.

Identification of Time-Varying Conceptual Hydrological Model Parameters with Differentiable Parameter Learning

“…Time-varying parameters have the potential to rectify the structural deficiencies inherent in traditional hydrological models of time-invariant parameters. Moreover, dPL had better performance than the empirical fm, with an increase in Corr and NSE by 0.055 and 0.114, respectively, in g Z, 6 . This suggests that dPL can extract more information from the relationships between parameters and inputs than the empirical fm.…”

“…Static attributes + P g Z, 3 Static attributes + T g Z, 4 Static attributes + SM g Z, 5 Static attributes + NDVI g Z, 6 Static attributes + PET + P + T + SM + NDVI…”

“…The GR4neige model integrates the GR4J model, which stands for the Génie Rural French model analyzing four parameters on a daily basis with the CemaNeige snow module. The GR4neige model is a lumped conceptual rainfall-runoff hydrological model and has found widespread applicability [5,6,49]. The model contains a simple model structure and merely six model parameters.…”

“…Conceptual hydrological models, such as the HBV model [1,2], GR4J model [3], and the Xinanjiang model [4], are employed for simulating and forecasting streamflow [5][6][7]. These models rely on parameters to formulate hypotheses regarding rainfall-runoff processes within a catchment [8,9].…”