Stochastic Optimal Control with Delay in the Control I: Solving the HJB Equation through Partial Smoothing

Abstract

Stochastic optimal control problems governed by delay equations with delay in the control are usually more difficult to study than the the ones when the delay appears only in the state. This is particularly true when we look at the associated Hamilton-Jacobi-Bellman (HJB) equation. Indeed, even in the simplified setting (introduced first by Vinter and Kwong [40] for the deterministic case) the HJB equation is an infinite dimensional second order semilinear Partial Differential Equation (PDE) that does not satisfy the so-called "structure condition" which substantially means that the control can act on the system modifying its dynamics at most along the same directions along which the noise acts. The absence of such condition, together with the lack of smoothing properties which is a common feature of problems with delay, prevents the use of the known techniques (based on Backward Stochastic Differential Equations (BSDEs) or on the smoothing properties of the linear part) to prove the existence of regular solutions of this HJB equation and so no results on this direction have been proved till now.In this paper we provide a result on existence of regular solutions of such kind of HJB equations. This opens the road to prove existence of optimal feedback controls, a task that will be accomplished in the companion paper [26]. The main tool used is a partial smoothing property that we prove for the transition semigroup associated to the uncontrolled problem. Such results hold for a specific class of equations and data which arises naturally in many applied problems. Key words:Optimal control of stochastic delay equations; Delay in the control; Lack of structure condition; Second order Hamilton-Jacobi-Bellman equations in infinite dimension; Smoothing properties of transition semigroups.

Stochastic Control with Delayed Information and Related Nonlinear Master Equation

“…A different aspect of delay in control problems is when the control chosen in a previous time, for instance at t − h, influences the dynamics and/or the cost function at time t. In the literature, this is usually called stochastic controls problems with delay in the control, see for example, Gozzi and Marinelli [2006], Gozzi and Masiero [2015], Alekal et al [1971], Chen and Wu [2011]. More generally, path dependence in the control was studied in Saporito [2017] in the framework of functional Itô calculus.…”

Verification theorems for stochastic optimal control problems in Hilbert spaces by means of a generalized Dynkin formula

“…The second example concerns the optimal control of a stochastic differential equation with delay in the control process (see [29,30] for the treatment of the same problem over finite horizon). In this case, the result we give needs to assume the existence of a mild solution to the associated elliptic HJB equation.…”

“…Here we consider an infinite horizon version of a control problem studied in [29,30]. Consider the following linear controlled one dimensional SDE: (6.26) where • W = {W(t)} t≥0 is a standard one dimensional Brownian motion;…”

“…A standard way to approach these delayed control problems, introduced in [47] for the deterministic case and extended to the stochastic case in [27], is to reformulate them as equivalent infinite dimensional control problems without delay 17 . The details are given in [29] for the finite horizon case, which is completely similar to the infinite horizon case, with the obvious changes (see also [17] for the infinite horizon case in a deterministic framework with a different embedding space). Consider the Hilbert space H := R× L 2 ([−d, 0], R), set b := (b 0 , b 1 (·)) ∈ H, and assume, without loss of generality, |b| H = 1.…”

“…In [29], the authors study a finite horizon optimal control problem with the same state equation (6.26) and a similar objective functional. Exploiting only partial smoothing properties of the transition semigroup associated to the state equation (6.28) with null control, the authors are able to provide, under suitable reasonable assumptions on the data, existence and uniqueness results for the parabolic HJB equation associated to the control problem.…”