Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space

Nüsken, Nikolas; Richter, Lorenz

doi:10.48550/arxiv.2005.05409

Cited by 10 publications

(21 citation statements)

References 90 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To the best of our knowledge, we are the first to establish a connection between IS and SOC in the context of pure jump processes, particularly for SRNs. Note that some existing works [26,30,28,27,36] have established a similar connection in the context of diffusion dynamics, mainly interested in efficiently estimating rare event probabilities using a path-dependent IS scheme.…”

Section: Introductionmentioning

confidence: 93%

Learning-Based Importance Sampling via Stochastic Optimal Control for Stochastic Reaction Networks

Hammouda¹,

Rached²,

Tempone³

et al. 2021

Preprint

View full text Add to dashboard Cite

Herein, we explore the efficient estimation of statistical quantities, particularly rare event probabilities, for stochastic reaction networks and biochemical systems. To this end, we propose a novel importance sampling (IS) approach to improve the efficiency of Monte Carlo (MC) estimators when based on an approximate tau-leap scheme. The crucial step in the IS framework is choosing an appropriate change of probability measure for achieving substantial variance reduction. Typically, this is challenging and often requires insights into the given problem. Based on an original connection between finding the optimal IS parameters within a class of probability measures and a stochastic optimal control (SOC) formulation, we propose an automated approach to obtain a highly efficient path-dependent measure change. The optimal IS parameters are obtained by solving a variance minimization problem. We begin by deriving an associated backward equation solved by these optimal parameters. Given the challenge of analytically solving this backward equation, we propose a numerical dynamic programming algorithm to approximate the optimal control parameters. In the one-dimensional case, our numerical results show that the variance of our proposed estimator decays at a rate of O(∆t) for a step size of ∆t, compared to O(1) for a standard MC estimator. For a given prescribed error tolerance, TOL, this implies an improvement in the computational complexity to become O(TOL −2 ) instead of O(TOL −3 ) when using a standard MC estimator. To mitigate the curse of dimensionality issue caused by solving the backward equation in the multi-dimensional case, we propose an alternative learning-based method that approximates the value function using a neural network, the parameters of which are determined via a stochastic optimization algorithm. The learning process uses the optimality criterion of our SOC problem, which relates the optimal controls to the value function. In this exploratory work, our numerical experiments demonstrate that our learning-based IS approach substantially reduces the variance of the MC estimator.

show abstract

Section: Introductionmentioning

confidence: 93%

Learning-Based Importance Sampling via Stochastic Optimal Control for Stochastic Reaction Networks

Hammouda¹,

Rached²,

Tempone³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Finally, in the deterministic limit the method becomes local as there is no diffusion to enforce the trajectories to explore the whole space. Similar techniques are applied in [26]- [28] based on different loss functions.…”

Section: B High-dimensional Stochastic Optimal Controlmentioning

confidence: 99%

A Neural Network Approach for High-Dimensional Optimal Control Applied to Multi-Agent Path Finding

Onken,

Nurbekyan,

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose a neural network approach for solving high-dimensional optimal control problems arising in real-time applications. Our approach yields controls in a feedback form and can therefore handle uncertainties such as perturbations to the system's state. We accomplish this by fusing the Pontryagin Maximum Principle (PMP) and Hamilton-Jacobi-Bellman (HJB) approaches and parameterizing the value function with a neural network. We train our neural network model using the objective function of the control problem and penalty terms that enforce the HJB equations. Therefore, our training algorithm does not involve data generated by another algorithm. By training on a distribution of initial states, we ensure the controls' optimality on a large portion of the state-space. Our grid-free approach scales efficiently to dimensions where grids become impractical or infeasible. We demonstrate the effectiveness of our approach on several multi-agent collisionavoidance problems in up to 150 dimensions. Furthermore, we empirically observe that the number of parameters in our approach scales linearly with the dimension of the control problem, thereby mitigating the curse of dimensionality.

show abstract

“…We now use these two ingredients to construct a connecting path between ρ 0 and ρ. By (56), there exists a T 1 < ∞ and a path ρ 1 : [0, T 1 ] → M connecting π to ρ 0 so that I [0,T1] (ρ 1 ) ≤ KL(ρ 0 )+1. By the time-reversal symmetry (57), the reversal ← − ρ 1 of this path connects ρ 0 to π, and satisfies I [0,T1] ( ← − ρ 1 ) ≤ 1.…”

Section: γ−Limmentioning

confidence: 99%

Stein Variational Gradient Descent: many-particle and long-time asymptotics

Nüsken¹,

Renger²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Stein variational gradient descent (SVGD) refers to a class of methods for Bayesian inference based on interacting particle systems. In this paper, we consider the originally proposed deterministic dynamics as well as a stochastic variant, each of which represent one of the two main paradigms in Bayesian computational statistics: variational inference and Markov chain Monte Carlo. As it turns out, these are tightly linked through a correspondence between gradient flow structures and large-deviation principles rooted in statistical physics. To expose this relationship, we develop the cotangent space construction for the Stein geometry, prove its basic properties, and determine the large-deviation functional governing the many-particle limit for the empirical measure. Moreover, we identify the Stein-Fisher information (or kernelised Stein discrepancy) as its leading order contribution in the long-time and many-particle regime in the sense of Γ-convergence, shedding some light on the finite-particle properties of SVGD. Finally, we establish a comparison principle between the Stein-Fisher information and RKHS-norms that might be of independent interest.

show abstract

Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space

Cited by 10 publications

References 90 publications

Learning-Based Importance Sampling via Stochastic Optimal Control for Stochastic Reaction Networks

Learning-Based Importance Sampling via Stochastic Optimal Control for Stochastic Reaction Networks

A Neural Network Approach for High-Dimensional Optimal Control Applied to Multi-Agent Path Finding

Stein Variational Gradient Descent: many-particle and long-time asymptotics

Contact Info

Product

Resources

About