Technical Note: Data assimilation and autoregression for using near-real-time streamflow observations in long short-term memory networks

Nearing, Grey; Klotz, Daniel; Sampson, Alden Keefe; Kratzert, Frederik; Gauch, Martin; Frame, Jonathan; Shalev, Guy; Nevo, Sella

doi:10.5194/hess-2021-515

Cited by 7 publications

(14 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, deep learning (DL) approaches have proven to be a promising tool in modeling hydrologic dynamics (Shen, 2018; Shen & Lawson, 2021; Sit et al., 2020). Among these, long short‐term memory (LSTM) networks (Hochreiter & Schmidhuber, 1997) present excellent performance in modeling soil moisture (Fang et al., 2017, 2019), streamflow (Feng et al., 2020; Frame et al., 2021; Gauch, Kratzert, et al., 2021; Ha et al., 2021; Kratzert et al., 2019; Nearing, Klotz, et al., 2021; Xiang & Demir, 2020), water table depth (Zhang et al., 2018), water quality variables such as water temperature (Rahmani et al., 2020, 2021) and dissolved oxygen (Zhi et al., 2021), and reservoir modulation (Ouyang et al., 2021). DL can be adapted for tasks like uncertainty quantification (Fang et al., 2020; Li et al., 2021), data assimilation (Fang & Shen, 2020; Feng et al., 2020), and multiscale modeling (Liu et al., 2022).…”

Section: Introductionmentioning

confidence: 99%

The Data Synergy Effects of Time‐Series Deep Learning Models in Hydrology

Fang

Kifer

Lawson

et al. 2022

Water Resources Research

View full text Add to dashboard Cite

When fitting statistical models to variables in geoscientific disciplines such as hydrology, it is a customary practice to stratify a large domain into multiple regions (or regimes) and study each region separately. Traditional wisdom suggests that models built for each region separately will have higher performance because of homogeneity within each region. However, each stratified model has access to fewer and less diverse data points. Here, through two hydrologic examples (soil moisture and streamflow), we show that conventional wisdom may no longer hold in the era of big data and deep learning (DL). We systematically examined an effect we call data synergy, where the results of the DL models improved when data were pooled together from characteristically different regions. The performance of the DL models benefited from modest diversity in the training data compared to a homogeneous training set, even with similar data quantity. Moreover, allowing heterogeneous training data makes eligible much larger training datasets, which is an inherent advantage of DL. A large, diverse data set is advantageous in terms of representing extreme events and future scenarios, which has strong implications for climate change impact assessment. The results here suggest the research community should place greater emphasis on data sharing.

show abstract

Section: Introductionmentioning

confidence: 99%

The Data Synergy Effects of Time‐Series Deep Learning Models in Hydrology

Fang

Kifer

Lawson

et al. 2022

Water Resources Research

View full text Add to dashboard Cite

show abstract

“…Many ML studies use readily available off‐the‐shelf products such as the catchment attributes and meteorology for large‐sample studies (CAMELS; Addor et al, 2017), which may impair their extensibility to practical, realistic applications where such data may not be available. Assimilation of new data into models using methods such as ensemble Kalman filters and autoregression (Brajard et al, 2020; Nearing, Klotz, et al, 2021; Zwart et al, 2021), and the use of integrated datasets tailored for the problem can improve prediction outcomes. Software that synthesize data for on‐demand queries such as brokering‐based tools (Horsburgh et al, 2016; Varadharajan et al, 2022), and methods to streamline quality control and outlier detection, gap‐fill, downscale observations, and determine parameters for process models (Bennett & Nijssen, 2021; Campbell et al, 2013; Hill & Minsker, 2010; Leigh et al, 2019; Mital et al, 2020; Russo et al, 2020) would ideally be integrated into ML workflows in parallel with advances in modelling approaches.…”

Section: Opportunities For Advancement Of Water Quality MLmentioning

confidence: 99%

Can machine learning accelerate process understanding and decision‐relevant predictions of river water quality?

Varadharajan

Appling

Arora

et al. 2022

Hydrological Processes

View full text Add to dashboard Cite

The global decline of water quality in rivers and streams has resulted in a pressing need to design new watershed management strategies. Water quality can be affected by multiple stressors including population growth, land use change, global warming, and extreme events, with repercussions on human and ecosystem health. A scientific understanding of factors affecting riverine water quality and predictions at local to regional scales, and at sub‐daily to decadal timescales are needed for optimal management of watersheds and river basins. Here, we discuss how machine learning (ML) can enable development of more accurate, computationally tractable, and scalable models for analysis and predictions of river water quality. We review relevant state‐of‐the art applications of ML for water quality models and discuss opportunities to improve the use of ML with emerging computational and mathematical methods for model selection, hyperparameter optimization, incorporating process knowledge into ML models, improving explainablity, uncertainty quantification, and model‐data integration. We then present considerations for using ML to address water quality problems given their scale and complexity, available data and computational resources, and stakeholder needs. When combined with decades of process understanding, interdisciplinary advances in knowledge‐guided ML, information theory, data integration, and analytics can help address fundamental science questions and enable decision‐relevant predictions of riverine water quality.

show abstract

“…the watershed). In addition to watershed delineations and precipitation estimates, they typically require data on both timevarying parameters (or forcing data) like temperature, humidity, soil moisture, and vegetation as well as static watershed properties like topography, soil type, or land use/land cover (Gauch et al, 2021;Kratzert et al, 2019Kratzert et al, , 2021Nearing et al, 2021). The rabpro API enables users to manage the complete data pipeline necessary to drive such a model starting from the initial watershed delineation through the calculation of static and time-varying parameters.…”

Section: Statement Of Needmentioning

confidence: 99%

rabpro: global watershed boundaries, river elevation profiles, and catchment statistics

Schwenk¹,

Zussman²,

Stachelek³

et al. 2022

JOSS

View full text Add to dashboard Cite

River and Basin Profiler (rabpro) is a Python package to delineate watersheds, extract river flowlines and elevation profiles, and compute watershed statistics for any location on the Earth's surface. As fundamental hydrologically-relevant units of surface area, watersheds are areas of land that drain via aboveground pathways to the same location, or outlet. Delineations of watershed boundaries are typically performed on digital elevation models (DEMs) that represent surface elevations as gridded rasters. Depending on the resolution of the DEM and the size of the watershed, delineation may be very computationally expensive. With this in mind, we designed rabpro to provide user-friendly workflows to manage the complexity and computational expense of watershed calculations given an arbitrary coordinate pair. In addition to basic watershed delineation, rabpro will extract the elevation profile for a watershed's mainchannel flowline. This enables the computation of river slope, which is a critical parameter in many hydrologic and geomorphologic models. Finally, rabpro provides a user-friendly wrapper around Google Earth Engine's (GEE) Python API to enable cloud-computing of zonal watershed statistics and/or time-varying forcing data from hundreds of available datasets. Altogether, rabpro provides the ability to automate or semi-automate complex watershed analysis workflows across broad spatial extents.

show abstract

Technical Note: Data assimilation and autoregression for using near-real-time streamflow observations in long short-term memory networks

Cited by 7 publications

References 28 publications

The Data Synergy Effects of Time‐Series Deep Learning Models in Hydrology

The Data Synergy Effects of Time‐Series Deep Learning Models in Hydrology

Can machine learning accelerate process understanding and decision‐relevant predictions of river water quality?

rabpro: global watershed boundaries, river elevation profiles, and catchment statistics

Contact Info

Product

Resources

About