Efficient Learning in Non-Stationary Linear Markov Decision Processes

Touati, Ahmed; Vincent, P.

doi:10.48550/arxiv.2010.12870

Cited by 9 publications

(20 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This matches the existing bounds in the general non-stationary linear kernel MDP setting without any constraints (Zhong et al, 2021;Zhou et al, 2020;Touati & Vincent, 2020). The dependence on the variation budgets (B ∆ , B ⋆ ) also matches the existing bound in policy-based method for the nonstationary linear kernel MDP setting (Zhong et al, 2021), but is worse than the Q-learning based method (Zhou et al, 2020;Touati & Vincent, 2020). However, Q-learning based method cannot solve CMDP since the optimal solution of CMDP is usually a stochastic policy.…”

Section: Resultssupporting

confidence: 88%

“…Periodically restarted optimistic policy evaluation Except for using the restart strategy (Mao et al, 2020;Zhou et al, 2020) to adapt to non-stationarity in the policy evaluation step, other strategies following the forgetting principle, such as sliding window (Zhong et al, 2021;Cheung et al, 2020) or exponential decayed weights (Touati & Vincent, 2020), are also expected to work in our framework.…”

Section: Discussionmentioning

confidence: 99%

“…Non-stationary RL. Non-stationary RL has been mostly studied in the unconstrained setting (Jaksch et al, 2010;Auer et al, 2019;Ortner et al, 2020;Domingues et al, 2021;Mao et al, 2020;Zhou et al, 2020;Touati & Vincent, 2020;Fei et al, 2020;Zhong et al, 2021;Cheung et al, 2020;Wei & Luo, 2021). Our work is related to policy-based methods for non-stationary RL since the optimal solution of CMDP is usually a stochastic policy (Altman, 1999) and thus a policy-based method is preferred.…”

Section: Related Workmentioning

confidence: 99%

“…From the recent work on the policy evaluation with UCB bonus (Jin et al, 2020;Zhou et al, 2020;Touati & Vincent, 2020), it can be shown that for every (m, h) ∈ [M ] × [H] and (x, a) ∈ S × A, with high probability, the cumulative model prediction error is bounded as…”

Section: Dynamic Regret Analysismentioning

confidence: 99%

See 3 more Smart Citations

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Ding¹,

Lavaei²

2022

Preprint

View full text Add to dashboard Cite

We consider primal-dual-based reinforcement learning (RL) in episodic constrained Markov decision processes (CMDPs) with non-stationary objectives and constraints, which play a central role in ensuring the safety of RL in timevarying environments. In this problem, the reward/utility functions and the state transition functions are both allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain known variation budgets. Designing safe RL algorithms in time-varying environments is particularly challenging because of the need to integrate the constraint violation reduction, safe exploration, and adaptation to the non-stationarity. To this end, we propose a Periodically Restarted Optimistic Primal-Dual Proximal Policy Optimization (PROPD-PPO) algorithm that features three mechanisms: periodic-restart-based policy improvement, dual update with dual regularization, and periodicrestart-based optimistic policy evaluation. We establish a dynamic regret bound and a constraint violation bound for the proposed algorithm in both the linear kernel CMDP function approximation setting and the tabular CMDP setting. This paper provides the first provably efficient algorithm for non-stationary CMDPs with safe exploration.

show abstract

Section: Resultssupporting

confidence: 88%

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Dynamic Regret Analysismentioning

confidence: 99%

See 2 more Smart Citations

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Ding¹,

Lavaei²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Although there is a huge body of literature on developing provably efficient RL methods, most the existing works focus on the classical stationary setting, with a few exceptions include Jaksch et al (2010); Gajane et al (2018); Cheung et al (2019aCheung et al ( ,c, 2020 Touati and Vincent (2020). However, these works all focus on value-based methods which only output greedy policies, and mostly focus on the tabular case where the state space is finite.…”

Section: Introductionmentioning

confidence: 99%

Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPs

Zhong¹,

Yang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

We study episodic reinforcement learning (RL) in non-stationary linear kernel Markov decision processes (MDPs). In this setting, both the reward function and the transition kernel are linear with respect to the given feature maps and are allowed to vary over time, as long as their respective parameter variations do not exceed certain variation budgets. We propose the periodically restarted optimistic policy optimization algorithm (PROPO), which is an optimistic policy optimization algorithm with linear function approximation. PROPO features two mechanisms: sliding-window-based policy evaluation and periodicrestart-based policy improvement, which are tailored for policy optimization in a non-stationary environment. In addition, only utilizing the technique of sliding window, we propose a value-iteration algorithm. We establish dynamic upper bounds for the proposed methods and a matching minimax lower bound which shows the (near-) optimality of the proposed methods. To our best knowledge, PROPO is the first provably efficient policy optimization algorithm that handles non-stationarity.

show abstract

Recent advances in reinforcement learning in finance

Hambly

Yang

2023

Mathematical Finance

View full text Add to dashboard Cite

The rapid changes in the finance industry due to the increasing amount of data have revolutionized the techniques on data processing and data analysis and brought new theoretical and computational challenges. In contrast to classical stochastic control theory and other analytical approaches for solving financial decision‐making problems that heavily reply on model assumptions, new developments from reinforcement learning (RL) are able to make full use of the large amount of financial data with fewer model assumptions and to improve decisions in complex financial environments. This survey paper aims to review the recent developments and use of RL approaches in finance. We give an introduction to Markov decision processes, which is the setting for many of the commonly used RL approaches. Various algorithms are then introduced with a focus on value‐ and policy‐based methods that do not require any model assumptions. Connections are made with neural networks to extend the framework to encompass deep RL algorithms. We then discuss in detail the application of these RL algorithms in a variety of decision‐making problems in finance, including optimal execution, portfolio optimization, option pricing and hedging, market making, smart order routing, and robo‐advising. Our survey concludes by pointing out a few possible future directions for research.

show abstract

Efficient Learning in Non-Stationary Linear Markov Decision Processes

Cited by 9 publications

References 10 publications

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPs

Recent advances in reinforcement learning in finance

Contact Info

Product

Resources

About