Maximum entropy optimal density control of discrete-time linear systems and Schrödinger bridges

Abstract

We consider an entropy-regularized version of optimal density control of deterministic discrete-time linear systems. Entropy regularization, or a maximum entropy (MaxEnt) method for optimal control has attracted much attention especially in reinforcement learning due to its many advantages such as a natural exploration strategy. Despite the merits, high-entropy control policies introduce probabilistic uncertainty into systems, which severely limits the applicability of MaxEnt optimal control to safety-critical systems. To remedy this situation, we impose a Gaussian density constraint at a specified time on the MaxEnt optimal control to directly control state uncertainty. Specifically, we derive the explicit form of the MaxEnt optimal density control. In addition, we also consider the case where a density constraint is replaced by a fixed point constraint. Then, we characterize the associated state process as a pinned process, which is a generalization of the Brownian bridge to linear systems. Finally, we reveal that the MaxEnt optimal density control induces the so-called Schr ödinger bridge associated to a discrete-time linear system.

Exact SDP Formulation for Discrete-Time Covariance Steering with Wasserstein Terminal Cost

“…Proof. The equalities µ k = µ k and v k = v k follow immediately given that the problem in (33) and the problem in (21) share the constraint in (12a) that enforces the state mean dynamics and in addition, the terms related to the mean dynamics in the objective functions J NLP and J SDP are equal. By the definition of M k in (24), it follows that tr…”

“…, N − 1}. In addition, the objective function J SDP is defined as follows: (21), and let {µ k , v k , Σ k , P k , M k , L} be the minimizer of the SDP in (33). Then,…”

“…Finally, the last term of the objective function of the NLP in (22) can be expressed as the optimal value of a relevant minimization problem in view of (3). Hence, the SDP problem in (33) is equivalent to the NLP problem in (21).…”

“…. , N − 1} and let π ∈ Π 2 be the optimal policy that solves the optimization problem in (21) and let {v k , K k , Q k } N −1 k=0 be its corresponding parameters. Then, Q k = 0 for all k ∈ {0, .…”

“…However, all of these approaches rely on a semi-definite relaxation of the terminal covariance constraint to reduce the CS problem to a convex program. Recently, a maximum entropy finite-horizon CS problem for discrete-time deterministic linear systems was addressed in [21] based on a Riccati equation approach.…”