Reliability Assessment of Machine Learning Models in Hydrological Predictions Through Metamorphic Testing

Abstract

The reliability of the machine learning model prediction for a given input can be assessed by comparing it against the actual output. However, in hydrological studies, machine learning models are often adopted to predict future or unknown events, where the actual outputs are unavailable. The prediction accuracy of a model, which measures its average performance across an observed data set, may not be relevant for a specific input. This study presents a method based on metamorphic testing (MT), adopted from software engineering, to assess the prediction reliability where the actual outputs are unknown. In this method, the predictions for a group of related inputs are considered consistent only if the input and output follow certain relations that are deduced from the properties of the system being modeled. For instance, the predicted runoff volume should increase in a rainfall‐runoff model as the rainfall magnitude of an input increases. In this study, the MT‐based method was applied to assess the predictions made by various machine learning models that were trained to predict the magnitude of flood events in Germany. Surprisingly, the prediction accuracy of a model and its ability to provide consistent predictions were found to be uncorrelated. This study further investigated the factors influencing the assessment result of a given input, such as its similarity to observed data. Overall, this research shows that MT is an effective and simple method for detecting inconsistent model predictions and is recommended when a model is employed to making predictions under new conditions.

Virtual Hydrological Laboratories: Developing the Next Generation of Conceptual Models to Support Decision Making Under Change

“…In a VHL, catchment characteristics can be assigned to cover a range of likely values, and VHL simulations used to create and test CMs that can then be more broadly applied. Testing the VHL could focus on behavioral modeling (Schaefli et al., 2011; Yang & Chui, 2021) to ensure that relationships between drivers of change and hydrologic responses are consistent with the four lines of evidence rather than using catchment specific performance metrics for single hydrologic variables (e.g., Nash Sutcliffe efficiency between observed and modeled flow) that rely heavily on having complete and accurate data sets to parameterize the model and define boundary conditions. Note that in this VHL approach, selection of suitable “trustworthy” PRMs for the particular change experiment of interest will be crucial—see Section 3.3 for further discussion.…”

Testing Ocean Software with Metamorphic Testing

“…In the hydraulic field, MT has been successfully applied to test the storm water management model (SWMM) systems [19] and dynamical hydraulic systems model [40]. Lin et al [19] have proposed four MRs related to the change of correlation coefficients ( 2 ), which help detect real-life defects in the SWMM system that is used to simulate the dynamic rainfall-runoff models for estimating the storm water runoffs in urban areas.…”

“…Lin et al [19] have proposed four MRs related to the change of correlation coefficients ( 2 ), which help detect real-life defects in the SWMM system that is used to simulate the dynamic rainfall-runoff models for estimating the storm water runoffs in urban areas. The work of Yang and Chui [40] focuses on testing machine learning models which predict flood events in Germany. They have proposed three MRs and adopted them to test 13 machine learning models.…”

“…For instances, in testing simulations, MT has been successfully adopted to test agent-based (ABM) and discrete-event (DES) simulations [26], hybrid ABM and DES systems [11], health care simulation [25], webenabled simulation [1], and simulator platform for self-driving cars [36,41]. In testing scientific software, MT is an effective technique to detect faults in simulation programs for designing nuclear power plants [14], bioinformatics programs [4], epidemiological models [31,33], chemical reaction networks for prototyping nano-scale molecular devices [13], matrix calculation programs [32], solvers for partial differential equations [3], multiple linear regression software [22], ocean modelling [16], storm water management model systems [19], machine learning-based hydro-logical models [40], Monte-Carlo computational programs [10,30], serverless scientific applications [20], as well as other types of scientific software [9,18,19,29]. For example, He et al [14] has found 33 bugs in simulation programs that are used to design and analyze nuclear power plants in a study that adopts MT.…”

Metamorphic Testing of Machine Learning and Conceptual Hydrologic Models

“…We will do this by considering isolated changes in precipitation and temperature and compare the outlet discharge with expected outcomes for selected basins from the CAMELS data set (Newman et al, 2015;Addor et al, 2017) with different characteristics (low-elevation, warm catchments vs. very high elevation, snow-dominated catchments). Note that this is a metamorphic testing design (Xie et al, 2011;Yang and Chui, 2021) that facilitates the formulation of the qualitative expected behavior, rather than a realistic climate change scenario that would consist of coupled temperature and precipitation changes. Based on this design, the more specific goals of our study are to answer the following questions:…”