Approximating the Stationary Bellman Equation by Hierarchical Tensor Products

Abstract

We treat infinite horizon optimal control problems by solving the associated stationary Hamilton-Jacobi-Bellman (HJB) equation numerically, for computing the value function and an optimal feedback area law. The dynamical systems under consideration are spatial discretizations of nonlinear parabolic partial differential equations (PDE), which means that the HJB is suffering from the curse of dimensions. To overcome numerical infeasability we use low-rank hierarchical tensor product approximation, or tree-based tensor formats, in particular tensor trains (TT tensors) and multipolynomials, since the resulting value function is expected to be smooth. To this end we reformulate the Policy Iteration algorithm as a linearization of HJB equations. The resulting linear hyperbolic PDE remains the computational bottleneck due to high-dimensions. By the methods of characteristics it can be reformulated via the Koopman operator in the spirit of dynamic programming. We use a low rank tensor representation for approximation of the value function. The resulting operator equation is solved using high-dimensional quadrature, e.g. Variational Monte-Carlo methods. From the knowledge of the value function at computable samples x i we infer the function x → v(x). We investigate the convergence of this procedure. By controlling destabilized versions of viscous Burgers and Schloegl equations numerical evidences are given.

Dynamical low-rank approximations of solutions to the Hamilton-Jacobi-Bellman equation

“…in ALS (compare to [OSS21b]). Here, c ∈ R rµ−1×nµ×rµ denotes the core that is currently optimised and .…”

“…We stress again that the version we used is the fastest version of the Bellman method available, since we employ the one-step scheme. As discussed in [OSS21a;OSS21b], this method also suffers from error propagation due to a large number of time steps. Moreover our proposed method projects onto the tangent space, whereas Bellman always tries to project onto the tensor manifold.…”

Particle Number Conservation and Block Structures in Matrix Product States

“…see, e.g., [2,7,8,23]. In this context, the bosonic particle number operator can be seen as a polynomial degree operator.…”

Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space

“…We also recommend [64] as an introduction to the specific setting considered in this paper. To cope with the key issue of high dimensionality, the authors of [83] suggest solving a certain type of control problem in the framework of hierarchical tensor products.…”