An Efficient DP Algorithm on a Tree-Structure for Finite Horizon Optimal Control Problems

Abstract

The classical Dynamic Programming (DP) approach to optimal control problems is based on the characterization of the value function as the unique viscosity solution of a Hamilton-Jacobi-Bellman (HJB) equation. The DP scheme for the numerical approximation of viscosity solutions of Bellman equations is typically based on a time discretization which is projected on a fixed state-space grid. The time discretization can be done by a one-step scheme for the dynamics and the projection on the grid typically uses a local interpolation. Clearly the use of a grid is a limitation with respect to possible applications in high-dimensional problems due to the curse of dimensionality.Here, we present a new approach for finite horizon optimal control problems where the value function is computed using a DP algorithm with a tree structure algorithm (TSA) constructed by the time discrete dynamics. In this way there is no need to build a fixed space triangulation and to project on it. The tree will guarantee a perfect matching with the discrete dynamics and drop off the cost of the space interpolation allowing for the solution of very high-dimensional problems. Numerical tests will show the effectiveness of the proposed method.

A hybrid stochastic river environmental restoration modeling with discrete and costly observations

“…The former dynamics may be observed almost completely, while the latter dynamics can be almost partly as considered in the presented model. Considering several state variables leads to an optimization problem having a huge degree of freedom whose numerical computation encounters the cure of dimensionality unless a sparse and efficient method is employed 97 . Considering both completely observable and partially observable dynamics would pose additional theoretical and technical issues.…”

Policy iteration for Hamilton–Jacobi–Bellman equations with control constraints

“…In the literature there are different techniques to tackle this situation. Related recent works include, polynomial approximation [28], deep neural technique [32], tensor calculus [18], Taylor series expansions [13] and graph-tree structures [5].…”

Robust Feedback Control of Nonlinear PDEs by Numerical Approximation of High-Dimensional Hamilton--Jacobi--Isaacs Equations

“…However, the design of numerical methods for the solution of high-dimensional HJ PDEs remains a daunting task. Along this direction, some encouraging results have been obtained over the last years in connection with the use of sparse grids [11,29], causality-free methods [34,22,48], machine learning techniques [40,39], tensor calculus [47], graph-tree structures [3], Taylor series expansions [17], and polynomial approximation [32]. In this latter work, we develop a numerical scheme based on a high-dimensional polynomial ansatz for the value function coupled with a Newton-type (policy iteration) method for the solution of the Galerkin residual equation associated to the HJB equation.…”