Newton’s Method, Bellman Recursion and Differential Dynamic Programming for Unconstrained Nonlinear Dynamic Games

Abstract

Dynamic games arise when multiple agents with differing objectives control a dynamic system. They model a wide variety of applications in economics, defense, energy systems and etc. However, compared to single-agent control problems, the computational methods for dynamic games are relatively limited. As in the single-agent case, only specific dynamic games can be solved exactly, so approximation algorithms are required. In this paper, we show how to extend the Newton step algorithm, the Bellman recursion and the popular differential dynamic programming (DDP) for single-agent optimal control to the case of fullinformation nonzero sum dynamic games. We show that the Newton's step can be solved in a computationally efficient manner and inherits its original quadratic convergence rate to open-loop Nash equilibria, and that the approximated Bellman recursion and DDP methods are very similar and can be used to find local feedback O(ε 2 )-Nash equilibria. Numerical examples are provided. KeywordsNoncooperative dynamic games • Open-loop Nash equilibrium • Feedback Nash equilibrium • Newton's method • Differential dynamic programming B Bolei Di

Stagewise Newton Method for Dynamic Game Control with Imperfect State Observation

“…While a naive implementation of the Newton method would yield a complexity of O(T 3 ), it is shown that the special structure of the Hessian induced by time can be exploited in order to obtain a linear complexity in time O(T ). In the perfect state observation case, [17] and [20] derived a stagewise Newton method with a backward recursion on the controls. However, with imperfect state observation, it is no longer clear how to do this with only one recursion.…”

“…For dynamic game control with perfect state information, the seminal work from [18], [19] introduced minimax DDP showing that DDP could be extended to zero-sum two players games. Recently, [20] further extended the concepts of stagewise Newton method and DDP to nonzero-sum games with an arbitrary number of players in the full information case.…”

Chance-Constrained Iterative Linear-Quadratic Stochastic Games

“…The key to this type of approach is to formulate the necessary conditions of open-loop Nash equilibria by concatenating the KKT conditions for each player. Di et al [14] solve the corresponding nonlinear program based on Newton's method. In [6], augmented Lagrangian is combined with Newton's method to handle state and control constraints.…”

“…Another limitation of the aforementioned approaches is that they only consider deterministic games and can not handle uncertainties in dynamics and observations. An increasingly popular branch of game-theoretic solvers is based on differential dynamic programming [4], [9], [14], [8]. The iterative linear-quadratic game method [4] exploits the analytical solutions of linear-quadratic games and solves the game as a sequence of approximated linear-quadratic games.…”

“…where q k is defined in (14). Since the covariance propagation is deterministic as we mentioned in Section IV.A, we can precompute the covariance before solving the game.…”