Connections between stochastic control and dynamic games

Abstract

We consider duality relations between risk-sensitive stochastic control problems and dynamic games. They are derived from two basic duality results, the first involving free energy and relative entropy and resulting from a Legendre-type transformation, the second involving power functions. Our approach allows us to treat, in essentially the same way, continuous- and discrete-time problems, with complete and partial state observation, and leads to a very natural formal justification of the structure of the cost functional of the dual. It also allows us to obtain the solution of a stochastic game problem by solving a risk-sensitive control problem

Relative Entropy, Exponential Utility, and Robust Dynamic Pricing

“…We emphasize, however, that the main focus of this paper is robust pricing, particularly the formulation of this problem using relative entropy and the computation and analysis of solutions to this problem, which we show to be no harder than that encountered in the classical problem formulated in Gallego and van Ryzin (1994). It is interesting to note, however, that the extension of this duality to more general robust pricing problems (such as multiproduct pricing) is significantly different from Charalambous et al (2004), Dai Pra et al (1996, Petersen et al (2000), and Ugrinovskii and Petersen (1999) and the results in this paper because multiproduct pricing involves the sharing of a common pool of resources by different products with ambiguous demands, and the constraint associated with resource sharing leads to interesting phenomena. The reader can consult Lim et al (2006) for more details.…”

“…The size of this constraint set represents our confidence in the nominal model, and in the special case where it is a singleton set, we recover the problem studied in Gallego and van Ryzin (1994). Similar approaches to the problem of dynamic optimization with uncertainty have been used in systems theory and electrical engineering (Charalambous et al 2004, Dai Pra et al 1996, Petersen et al 2000, Ugrinovskii and Petersen 1999, 2001) as well as economics and finance (Hansen et al 2005), although in these contexts, randomness is modelled using Brownian motion. One contribution of this paper is the application of this general relative entropy approach to the problem of dynamic revenue management where uncertainty is modelled by a point process.…”

“…In particular, the value function of the robust problem can be interpreted as the certainty equivalent of the value function of the problem with an exponential utility, and the optimal pricing policies for both problems are shown to be the same. The equivalence is actually a special case of the duality between free energy and relative entropy from the theory of large deviations (Dupuis and Ellis 1997) that has been exploited in the area of risk-sensitive control (Dai Pra et al 1996, Petersen et al 2000. For the sake of completeness, we derive this correspondence via the associated optimality equations.…”

“…This formulation (an adaptation of ideas from Dai Pra et al 1996, Petersen et al 2000, and Hansen et al 2005) allows us to account for possible path dependence in the actual real-world arrival rate process and errors in the nominal model through a constraint on relative entropy. We show in §4 how the problem of optimal pricing when there is model uncertainty can be formulated as a stochastic differential game (see Dai Pra et al 1996, Petersen et al 2000, and solved through a version of the so-called Isaacs' equation, which is a generalization of the dynamic programming equation to two-player zero-sum stochastic differential games. The Isaacs' equation is a nonlinear ordinary differential equation (ODE), and the focus of §4.1 is to address certain fundamental technical questions associated with this equation.…”

Robustness and information processing

“…In particular, we shall use the version in (23) with ψ ≡ (X s −X s ) 2 to replace the 'inner part' of the minmax filtering problem with a conventional risk-sensitive objective function. Although the ψ function in our case is quadratic, and hence unbounded, Dai Pra, Meneghini, and Runggaldier (1996) show that the duality in lemma 3.1 can be extended to cover this case as well. Solving the resulting risk-sensitive filtering problem yields, …”