A penalty scheme and policy iteration for nonlocal HJB variational inequalities with monotone drivers

Abstract

We propose a class of numerical schemes for nonlocal HJB variational inequalities (HJBVIs) with monotone drivers. The solution and free boundary of the HJBVI are constructed from a sequence of penalized equations, for which a continuous dependence result is derived and the penalization error is estimated. The penalized equation is then discretized by a class of semiimplicit monotone approximations. We present a novel analysis technique for the well-posedness of the discrete equation, and demonstrate the convergence of the scheme, which subsequently gives a constructive proof for the existence of a solution to the penalized equation and variational inequality. We further propose an efficient iterative algorithm with local superlinear convergence for solving the discrete equation. Numerical experiments are presented for an optimal investment problem under ambiguity and a recursive consumption-portfolio allocation problem.

A neural network based policy iteration algorithm with global $H^2$-superlinear convergence for stochastic games on domains

“…We start by presenting the policy iteration scheme for the HJBI equations in Algorithm 1, which extends the policy iteration algorithm (or Howard's algorithm) for discrete HJB equations (see e.g. [14,5,34]) to the continuous setting.…”

“…Finally, we remark that the proof of superlinear convergence of our algorithm is significantly different from the arguments for discrete equations. Instead of working with the sup-norm for (finite-dimensional) discrete equations as in [37,14,5,42,34], we employ a two-norm framework to establish the generalized differentiability of HJBI operators, where the norm gap is essential as has already been pointed out in [20,41,40]. Moreover, by taking advantage of the fact that the Hamiltonian only involves low order terms, we further demonstrate that the inverse of the generalized derivative is uniformly bounded.…”

“…A nonconvex HJBI equation as above also arises from a penalty approximation of hybrid control problems involving continous controls, optimal stopping and impulse controls, where the HJB (quasi-)variational inequality can be reduced to an HJBI equation by penalizing the difference between the value function and the obstacles (see e.g. [23,42,34,35]). As (1.1) in general cannot be solved analytically, it is important to construct effective numerical schemes to find the solution of (1.1) and its derivatives.…”

“…The standard approach to solving (1.1) is to "discretize, then optimize", where one first discretizes the operators in (1.1) by finite difference or finite element methods, and then solves the resulting nonlinear discretized equations by using policy iteration, also known as Howard's algorithm, or generally (finite-dimensional) semismooth Newton methods (see e.g. [14,5,40,34]). However, this approach has the following drawbacks, as do most mesh-based methods: (1) it can be difficult to generate meshes and to construct consistent numerical schemes for problems in domains with complicated geometries; (2) the number of unknowns in general grows exponentially with the dimension n, i.e., it suffers from Bellman's curse of dimensionality, and hence this approach is infeasible for solving high-dimensional control problems.…”

“…Although the (local) superlinear convergence of policy iteration for discrete equations has been proved in various works (e.g. [33,37,14,5,42,40,34]), to the best of our knowledge, there is no published work on the superlinear convergence of policy iteration for HJB PDEs in a function space, nor on the global convergence of policy iteration for solving nonconvex HJBI equations.…”