Bayesian LSTM With Stochastic Variational Inference for Estimating Model Uncertainty in Process‐Based Hydrological Models

Abstract

Significant attention has recently been paid to deep learning as a method for improved catchment modeling. Compared with process‐based models, deep learning is often criticized for its lack of interpretability. One solution is to combine a process‐based hydrological model with a residual error model based on deep learning to give full scope to their respective advantages. In classical residual error models, Bayesian inference via Markov chain Monte Carlo (MCMC) is commonly used to provide an estimation of the uncertainty. However, deep neural networks tend to have excessively large numbers of parameters, making MCMC an unsuitable approach. Here, we introduce an alternative to Bayesian MCMC sampling called stochastic variational inference (SVI) which has recently been developed for Bayesian deep learning in Natural Language Processing. We implement SVI in a Long Short‐Term Memory (LSTM) network and construct residual error models in process‐based hydrological models. This approach is examined in the contrasting geographical and climatic characteristics of two catchments from China, the Tangnaihai catchment and the Shiquan catchment. Compared with the Bayesian linear regression model, the Bayesian LSTM provides better uncertainty estimates. Specifically, the proposed method improves the Continuous Ranked Probability Score (CRPS) by over 10% in both two catchments. In the Tangnaihai catchment, it provides more than 10% narrower uncertainty intervals in terms of Sharpness with slightly superior Reliability. In the Shiquan catchment, it provides comparable uncertainty intervals with better Reliability. Further, our study highlights the scalability of SVI to high‐dimensional parameter spaces in hydrological applications (e.g., distributed hydrological models, groundwater models).

Observation‐Constrained Projection of Flood Risks and Socioeconomic Exposure in China

“…In the last two decades, support vector machines (SVM; Lin et al., 2009), artificial neural networks (ANN; M. Cheng et al., 2020) and Long Short‐Term Memory (LSTM; Lee et al., 2020; Lees et al., 2022) have been utilized to strengthen hydrological predictions by using large‐sample (multi‐site) data. LSTMs in particular can achieve promising results, provide insights into hydrological processes (Lees et al., 2022), and reduce uncertainty in projections of future flood dynamics (D. Y. Li et al., 2021; W. B. Liu et al., 2021).…”

Recurrent neural networks enable design of multifunctional synthetic human gut microbiome dynamics

“…The current implementation of the LSTM model lacks uncertainty quantification for individual predictions, which could be used to guide experimental design ( Radivojević et al, 2020 ). Recent progress in using Bayesian recurrent neural networks has led to emergence of Bayesian LSTMs ( Fortunato et al, 2017 ; Li et al, 2021 ), which provides uncertainty quantification for each prediction in the form of posterior variance or posterior confidence interval. However, currently, the implementation and training of such Bayesian neural networks can be significantly more difficult than training the LSTM model developed here.…”

Machine Learning for Bayesian Experimental Design in the Subsurface

“…Only a few regression techniques allow for direct probabilistic predictions (e.g., Gaussian Process Regression). However, some ML algorithms used for (deterministic) regression, such as gradient boosting, neural networks, and Long Short-Term Memory (LSTM) networks, have been modified, hybridized, or extended to provide probabilistic predictions thereby providing a mechanism to quantify the uncertainty (e.g., Dillon et al 2017;Duan et al 2019;Klotz et al 2022;Li et al 2021).…”

“…Lees et al (2022) and Nearing et al (2022) investigated the information captured by the LSTM state vector and compared two approaches for ingesting near-real-time streamflow observations for rainfall-runoff modeling. Li et al (2021) suggested using Bayesian LSTM with stochastic variational inference to estimate model uncertainty in process-based hydrological models. Klotz et al (2022) trained their models by maximizing the log-likelihood function of the observations according to the predicted mixture distributions, and benchmarked different model setups.…”