An Online Actor–Critic Algorithm with Function Approximation for Constrained Markov Decision Processes

In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such criteria is known to be a hard problem. In this paper, we consider both discounted and average reward Markov decision processes. For each formulation, we first define a measure of variability for a policy, which in turn gives us a set of risk-sensitive criteria to optimize. For each of these criteria, we derive a formula for computing its gradient. We then devise actor-critic algorithms that operate on three timescales-a TD critic on the fastest timescale, a policy gradient (actor) on the intermediate timescale, and a dual ascent for Lagrange multipliers on the slowest timescale. In the discounted setting, we point out the difficulty in estimating the gradient of the variance of the return and incorporate simultaneous perturbation approaches to alleviate this. The average setting, on the other hand, allows for an actor update using compatible features to estimate the gradient of the variance. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in a traffic signal control application.

show abstract

“…Proof The proof follows in a similar manner as that of Theorem 3 in Bhatnagar and Lakshmanan (2012).…”

Section: Convergence Analysis Of the Average Reward Risk-sensitive Acmentioning

confidence: 89%

Variance-constrained actor-critic algorithms for discounted and average reward MDPs

Prashanth

Ghavamzadeh

2016

Mach Learn

View full text Add to dashboard Cite

show abstract

“…While most methods for solving constrained MDPs revolve around the use of mathematical programs, some reinforcement learning approaches have also been proposed for optimizing the average-reward objective and, to a lesser extent, for solving constrained instances of average-reward MDPs. Some noteworthy examples include the constrained actor-critic method proposed by Bhatnagar and Lakshmanan (2012), wherein a Lagrangian relaxation of the problem is used to incorporate steady-state costs into the objective function being optimized by the constrained actor-critic algorithm. A similar Lagrangian Q-learning approach is proposed by Lakshmanan and Bhatnagar (2012).…”

Section: Related Workmentioning

confidence: 99%

Steady-State Planning in Expected Reward Multichain MDPs

Atia¹,

Beckus²,

Alkhouri³

et al. 2021

jair

View full text Add to dashboard Cite

The planning domain has experienced increased interest in the formal synthesis of decision-making policies. This formal synthesis typically entails finding a policy which satisfies formal specifications in the form of some well-defined logic. While many such logics have been proposed with varying degrees of expressiveness and complexity in their capacity to capture desirable agent behavior, their value is limited when deriving decision-making policies which satisfy certain types of asymptotic behavior in general system models. In particular, we are interested in specifying constraints on the steady-state behavior of an agent, which captures the proportion of time an agent spends in each state as it interacts for an indefinite period of time with its environment. This is sometimes called the average or expected behavior of the agent and the associated planning problem is faced with significant challenges unless strong restrictions are imposed on the underlying model in terms of the connectivity of its graph structure. In this paper, we explore this steady-state planning problem that consists of deriving a decision-making policy for an agent such that constraints on its steady-state behavior are satisfied. A linear programming solution for the general case of multichain Markov Decision Processes (MDPs) is proposed and we prove that optimal solutions to the proposed programs yield stationary policies with rigorous guarantees of behavior.

show abstract

“…Other Settings. We also want to mention some related literature in game theory [39,40,41,42,43], two-time-scale stochastic approximation [44,45,46,47,48,49,50,51,52,53], reinforcement learning [54,55,56,57,58], two-time-scale optimization [59,60], and decentralized optimization [61,62,63,64,65,66,67]. These works study different variants of two-time-scale methods mostly for solving a single optimization problem, and often aim to find global optimality (or fixed points) using different structure of the underlying problems (e.g., Markov structure in stochastic games and reinforcement learning or strong monotonicity in stochastic approximation).…”

Section: Related Workmentioning

confidence: 99%

Convergence Rates of Two-Time-Scale Gradient Descent-Ascent Dynamics for Solving Nonconvex Min-Max Problems

Doan¹

2021

Preprint

View full text Add to dashboard Cite

There are much recent interests in solving noncovnex min-max optimization problems due to its broad applications in many areas including machine learning, networked resource allocations, and distributed optimization. Perhaps, the most popular first-order method in solving min-max optimization is the so-called simultaneous (or single-loop) gradient descent-ascent algorithm due to its simplicity in implementation. However, theoretical guarantees on the convergence of this algorithm is very sparse since it can diverge even in a simple bilinear problem.In this paper, our focus is to characterize the finite-time performance (or convergence rates) of the continuous-time variant of simultaneous gradient descent-ascent algorithm. In particular, we derive the rates of convergence of this method under a number of different conditions on the underlying objective function, namely, two-sided Polyak-Lojasiewicz (P L), one-sided P L, nonconvex-strongly concave, and strongly convex-nonconcave conditions. Our convergence results improve the ones in prior works under the same conditions of objective functions. The key idea in our analysis is to use the classic singular perturbation theory and coupling Lyapunov functions to address the time-scale difference and interactions between the gradient descent and ascent dynamics. Our results on the behavior of continuous-time algorithm may be used to enhance the convergence properties of its discrete-time counterpart.

show abstract

An Online Actor–Critic Algorithm with Function Approximation for Constrained Markov Decision Processes

Cited by 66 publications

References 16 publications

Variance-constrained actor-critic algorithms for discounted and average reward MDPs

Variance-constrained actor-critic algorithms for discounted and average reward MDPs

Steady-State Planning in Expected Reward Multichain MDPs

Convergence Rates of Two-Time-Scale Gradient Descent-Ascent Dynamics for Solving Nonconvex Min-Max Problems

Contact Info

Product

Resources

About