DAMP: A protocol for contextualising goodness-of-fit statistics in sediment-discharge data-driven modelling

Abstract

The decision sequence which guides the selection of a preferred data-driven modelling solution is usually based solely on statistical assessment of fit to a test dataset, and lacks the incorporation of essential contextual knowledge and understanding included in the evaluation of conventional empirical models. This paper demonstrates how hydrological insight and knowledge of data quality issues can be better incorporated into the sediment-discharge data-driven model assessment procedure: by the plotting of datasets and modelled relationships; and from an understanding and appreciation of the hydrological context of the catchment being modelled. DAMP: a four-point protocol for evaluating the hydrological soundness of data-driven single-input single-output sediment rating curve solutions is presented. The approach is adopted and exemplified in an evaluation of seven explicit sediment-discharge models that are used to predict daily suspended sediment concentration values for a small tropical catchment on the island of Puerto Rico. Four neurocomputing counterparts are compared and contrasted against a set of traditional log-log linear sediment rating curve solutions and a simple linear regression model. The statistical assessment procedure provides one indication of the best model, whilst graphical and hydrological interpretation of the depicted datasets and models challenge this overly-simplistic interpretation. Traditional log-log sediment rating curves, in terms of soundness and robustness, are found to deliver a superior overall product -irrespective of their poorer global goodness-of-fit statistics.

Legitimising data-driven models: exemplification of a new data-driven mechanistic modelling framework

“…Shrestha and Nestmann, 2009). However, an important question remains about whether they can ever offer more than the optimisation of goodness-of-fit between inputs and outputs through the delivery of insights to hydrologists (Minns and Hall, 1996;Babovic, 2005;Abrahart et al, 2011). This question is particularly pertinent for ANN-based models, which represent the most widely used type of a black-box DDM in hydrology.…”

“…Indeed, DDMs in general, and ANN-based models in particular, have been criticised as being little more than advanced curve-fitting tools with limited heuristic value (e.g. Abrahart et al, 2011). To those engaged in DDM and ANN-based modelling, this view can seem intuitively wrong.…”

Including spatial distribution in a data-driven rainfall-runoff model to improve reservoir inflow forecasting in Taiwan

“…However, stronger calls for greater incorporation of scientific knowledge and understanding in the development of data‐driven hydrological models are now starting to be published (e.g. Abrahart et al ., ) in which it is argued that better representation of catchment processes should result in improved data‐driven modelling products that offer more than optimised curve fitting solutions (Mount and Abrahart, ).…”

The need for operational reasoning in data‐driven rating curve prediction of suspended sediment

“…as concentration or load/logged or unlogged) and the impact that decision has on the form and consistency of outputs. In Abrahart et al (), we examine how contextual hydrological and dataset knowledge can be incorporated into the process of developing and selecting data‐driven suspended sediment models and call for an end to blind model justification on the sole basis of goodness‐of‐fit metrics.…”

“…Indeed, papers assessing the relative performance of different data‐driven algorithms, techniques and model configurations under different modelling scenarios are far more numerous than those which examine the influence that the modeller's reasoning processes may be having on the usefulness or correctness of their assessments, or the physical interpretation of their results – issues that have received far more attention from physically based modellers ( cf Beven, ). As a consequence, a number of key criticisms of the data‐driven paradigm that apply to suspended sediment modelling remain largely unaddressed including a general lack of justification for the model configuration and response function structures used (Minns and Hall, ; Babovic, ; Solomatine et al , ); lack of established procedures to ensure robust model configurations and outputs (Abrahart et al , ; Mount and Abrahart, ); lack of justification for using more complex, data‐driven response functions over simpler empirical counterparts (Mount and Abrahart, ) over‐reliance on simplistic, goodness‐of‐fit metrics in the identification of preferred models (Legates and McCabe, ; Abrahart et al , ). lack of operational applications for the model, i.e. can the resultant model actually be applied for an operational purpose?…”