Differentiable modelling to unify machine learning and physical models for geosciences

Appling, Alison P.; Gentine, Pierre; Bandai, Toshiyuki; Gupta, Hoshin; Tartakovsky, Alexandre M.; Baity‐Jesi, Marco; Fenicia, Fabrizio; Kifer, Daniel; Li, Li; Liu, Xiaofeng; Ren, Wei; Zheng, Yi; Harman, C. J.; Clark, Martyn P.; Farthing, Matthew W.; Feng, Dapeng; Prabhash, Kumar; Aboelyazeed, Doaa; Rahmani, Farshid; Song, Yalan; Beck, Hylke E.; Bindas, Tadd; Dwivedi, Dipankar; Fang, Kuai; Höge, Marvin; Rackauckas, Chris; Mohanty, Binayak P.; Roy, Tirthankar; Xu, Chonggang; Lawson, Kathryn

doi:10.1038/s43017-023-00450-9

Cited by 81 publications

(47 citation statements)

References 152 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DS are not strictly necessary because a neural network may be trained 'offline' and then implemented only using a forward pass in a physics simulator. However, DS enable more flexibility in the set of equations that contain symbolic expressions and neural networks which the user wishes to solve [19,20]. We exploit this advantage in this work because we seek to implement an additional set of equations for the evolution of a quantity that has no corresponding observables in the training data, that is they are 'hidden' .…”

Section: Inclusion Of Kinetic Effects In a Fluid Codementioning

confidence: 99%

Machine learning of hidden variables in multiscale fluid simulation

Joglekar,

Thomas

2023

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

Solving fluid dynamics equations often requires the use of closure relations that account for missing microphysics. For example, when solving equations related to fluid dynamics for systems with a large Reynolds number, sub-grid effects become important and a turbulence closure is required, and in systems with a large Knudsen number, kinetic effects become important and a kinetic closure is required. By adding an equation governing the growth and transport of the quantity requiring the closure relation, it becomes possible to capture microphysics through the introduction of ``hidden variables'' that are non-local in space and time. The behavior of the ``hidden variables'' in response to the fluid conditions can be learned from a higher fidelity or ab-initio model that contains all the microphysics. In our study, a partial differential equation simulator that is end-to-end differentiable is used to train judiciously placed neural networks against ground-truth simulations. We show that this method enables an Euler equation based approach to reproduce non-linear, large Knudsen number plasma physics that can otherwise only be modeled using Boltzmann-like equation simulators such as Vlasov or Particle-In-Cell modeling.

show abstract

Section: Inclusion Of Kinetic Effects In a Fluid Codementioning

confidence: 99%

Machine learning of hidden variables in multiscale fluid simulation

Joglekar,

Thomas

2023

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

show abstract

“…This work demonstrates the advantage of differentiable modeling-the genre of model that allows gradient information to be propagated along the entire chain of calculations and thus support training of NNs anywhere in the model. We recommend that readers refer to Shen et al (2023) and Tsai et al (2021) for a more comprehensive discussion. Traditional calibration or response-surface approaches typically work on a site-by-site basis and cannot train a massive amount of weights in complex NNs on large data, and thus typically fall into the problem of non-uniqueness, also called equifinality.…”

Section: Further Modeling Implicationsmentioning

confidence: 99%

“…To overcome DL’s limitations while benefiting from its ability to learn from big data, a new class of physics‐informed machine learning models—“differentiable models”—has emerged. They harness the core technology behind DL, differentiable programming, while including process‐based equations as model priors or constraints of the system (Shen et al., 2023). These models enable efficient and accurate calculations of gradients of the outputs with respect to the variables used in the model, and these gradients are used to update weights in the connected neural networks (NNs) or parameters in the model.…”

Section: Introductionmentioning

confidence: 99%

“…The untrained variables (those that were not provided as training targets but are calculated and output due to the physical priors) like evapotranspiration also compared well with alternative estimates (Feng et al., 2022). Differentiable models leverage the advantages of speed and scale (parameters can be rapidly estimated for many sites at once) offered by modern artificial intelligence development (Shen et al., 2023).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Identifying Structural Priors in a Hybrid Differentiable Model for Stream Water Temperature Modeling

Rahmani,

Appling,

Feng

et al. 2023

Water Resources Research

Self Cite

View full text Add to dashboard Cite

Although deep learning models for stream temperature (Ts) have recently shown exceptional accuracy, they have limited interpretability and cannot output untrained variables. With hybrid differentiable models, neural networks (NNs) can be connected to physically based equations (called structural priors) to output intermediate variables such as water source fractions (specifying what portion of water is groundwater, subsurface, and surface flow). However, it is unclear if such outputs are physically meaningful when only limited physics is imposed, and if structural priors have enough impacts to be identifiable from data. Here, we tested four alternative structural priors describing basin‐scale water temperature memory and instream heat processes in a differentiable stream temperature model where NNs freely estimate the water source fractions. We evaluated models’ abilities to predict Ts and baseflow ratio. The four priors exhibited noticeably different behaviors in these two metrics and their tradeoffs, with some dominating others. Therefore, the better structural priors can be identified. Moreover, testing different priors yielded valuable insights: having a separate shallow subsurface flow component better matches observations, and a recency‐weighted averaging of past air temperature for calculating source water temperature resulted in better Ts and baseflow prediction than traditionally employed simple averaging. However, we also highlight the limitations when insufficient physical constraints are implemented: the internal variables (water source fractions) may not be adequately constrained by a single target variable (stream temperature) alone. To ensure the physical significance of the internal fluxes, one can either employ multivariate data for model selection, or include more physical processes in the priors.

show abstract

“…Additional custom loss terms would be possible and could make use of other data, such as observed groundwater temperatures and levels, if a differentiable model were used that represented those intermediate variables within process-based equations (Shen et al, 2023). In the DRB, there were only two sites with groundwater wells with daily water temperature observations (all occurring within the test partition) and only 24 wells with more than 20 discrete groundwater temperature observations.…”

Section: Implementation Considerationsmentioning

confidence: 99%

Train, Inform, Borrow, or Combine? Approaches to Process‐Guided Deep Learning for Groundwater‐Influenced Stream Temperature Prediction

Barclay,

Topp,

Koenig

et al. 2023

Water Resources Research

Self Cite

View full text Add to dashboard Cite

Although groundwater discharge is a critical stream temperature control process, it is not explicitly represented in many stream temperature models, an omission that may reduce predictive accuracy, hinder management of aquatic habitat, and decrease user confidence. We assessed the performance of a previously‐described process‐guided deep learning model of stream temperature in the Delaware River Basin (USA). We found lower accuracy (root mean square error [RMSE] of 1.71 versus 1.35°C) and stronger seasonal bias (absolute mean monthly bias of 1.06 vs. 0.68°C) for reaches primarily influenced by deep groundwater as compared to atmospheric conditions. We then tested four approaches for improving groundwater process representation: (a) a custom loss function leveraging the unique patterns of air and water temperature coupling characteristic of different temperature drivers, (b) inclusion of additional groundwater‐relevant catchment attributes, (c) incorporation of additional process model outputs, and (d) a composite model. The custom loss function and the additional attributes significantly improved the predictive accuracy in groundwater‐dominated reaches (RMSE of 1.37 and 1.26°C) and reduced the seasonal bias (absolute mean monthly bias of 0.44 and 0.48°C), but neither approach could identify holdout groundwater reaches. Variable importance analysis indicates the custom loss function nudges the model to use the existing inputs more efficiently, whereas with the added features the model relies on a broader suite of inputs. This analysis is a substantial step toward more accurately representing groundwater discharge processes in stream temperature models and will improve predictive accuracy and inform habitat management.

show abstract

Differentiable modelling to unify machine learning and physical models for geosciences

Cited by 81 publications

References 152 publications

Machine learning of hidden variables in multiscale fluid simulation

Machine learning of hidden variables in multiscale fluid simulation

Identifying Structural Priors in a Hybrid Differentiable Model for Stream Water Temperature Modeling

Train, Inform, Borrow, or Combine? Approaches to Process‐Guided Deep Learning for Groundwater‐Influenced Stream Temperature Prediction

Contact Info

Product

Resources

About