Challenges of Real-World Reinforcement Learning

Dulac-Arnold, Gabriel; Mankowitz, Daniel J.; Hester, Todd

doi:10.48550/arxiv.1904.12901

Cited by 114 publications

(149 citation statements)

References 26 publications

Supporting

Mentioning

136

Contrasting

Order By: Relevance

“…CMDP. The study of RL algorithms for CMDPs has received considerable attention due to the safety requirement (Altman, 1999;Paternain et al, 2019;Yu et al, 2019;Dulac-Arnold et al, 2019;Garcıa & Fernández, 2015). Our work is closely related to Lagrangian-based CMDP algorithms with optimistic policy evaluations (Efroni et al, 2020;Singh et al, 2020;Ding et al, 2021;Liu et al, 2021;Qiu et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

“…Safe reinforcement learning (RL) studies how an agent learns to maximize its expected total reward by interacting with an unknown environment over time while dealing with restrictions/constraints arising from real-world problems (Amodei et al, 2016;Dulac-Arnold et al, 2019;Garcıa & Fernández, 2015). A standard approach for mod-1 Department of Industrial Engineering and Operations Research, University of California, Berkeley (e-mail: yuhao_ding@berkeley.edu); 2 Department of Industrial Engineering and Operations Research, University of California, Berkeley (e-mail: lavaei@berkeley.edu).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Ding¹,

Lavaei²

2022

Preprint

View full text Add to dashboard Cite

We consider primal-dual-based reinforcement learning (RL) in episodic constrained Markov decision processes (CMDPs) with non-stationary objectives and constraints, which play a central role in ensuring the safety of RL in timevarying environments. In this problem, the reward/utility functions and the state transition functions are both allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain known variation budgets. Designing safe RL algorithms in time-varying environments is particularly challenging because of the need to integrate the constraint violation reduction, safe exploration, and adaptation to the non-stationarity. To this end, we propose a Periodically Restarted Optimistic Primal-Dual Proximal Policy Optimization (PROPD-PPO) algorithm that features three mechanisms: periodic-restart-based policy improvement, dual update with dual regularization, and periodicrestart-based optimistic policy evaluation. We establish a dynamic regret bound and a constraint violation bound for the proposed algorithm in both the linear kernel CMDP function approximation setting and the tabular CMDP setting. This paper provides the first provably efficient algorithm for non-stationary CMDPs with safe exploration.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Ding¹,

Lavaei²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…An important limitation of DRL methods is their sample inefficiency: an enormous amount of data is necessary and makes training expensive. This makes applying DRL in the real world challenging, for example in robotics (Sünderhauf et al, 2018;Dulac-Arnold et al, 2019). In tasks like manipulation, sample collection is a slow and costly process (Liu et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation

Mai¹,

Mani²,

Paull³

2022

Preprint

View full text Add to dashboard Cite

In model-free deep reinforcement learning (RL) algorithms, using noisy value estimates to supervise policy evaluation and optimization is detrimental to the sample efficiency. As this noise is heteroscedastic, its effects can be mitigated using uncertainty-based weights in the optimization process. Previous methods rely on sampled ensembles, which do not capture all aspects of uncertainty. We provide a systematic analysis of the sources of uncertainty in the noisy supervision that occurs in RL, and introduce inverse-variance RL, a Bayesian framework which combines probabilistic ensembles and Batch Inverse Variance weighting. We propose a method whereby two complementary uncertainty estimation methods account for both the Q-value and the environment stochasticity to better mitigate the negative impacts of noisy supervision. Our results show significant improvement in terms of sample efficiency on discrete and continuous control tasks.

show abstract

“…The design of adequate reward functions poses a tremendous challenge for building reinforcement learning (RL) agents that ought to act in accordance with human intentions [4,13]. Besides complicating the deployment of RL in the real world [11], this can lead to major unforeseen societal impacts, which need to be accounted for when building autonomous systems [6,45]. To tackle this, the field of value alignment has largely focused on reward learning, which aims to adopt a bottom-up approach of finding goal specifications from observational data instead of manually specifying them [22,30,40].…”

Section: Introductionmentioning

confidence: 99%

MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning

Peschl¹,

Zgonnikov²,

Oliehoek³

et al. 2022

Preprint

View full text Add to dashboard Cite

Inferring reward functions from demonstrations and pairwise preferences are auspicious approaches for aligning Reinforcement Learning (RL) agents with human intentions. However, state-of-the art methods typically focus on learning a single reward model, thus rendering it difficult to trade off different reward functions from multiple experts. We propose Multi-Objective Reinforced Active Learning (MORAL), a novel method for combining diverse demonstrations of social norms into a Pareto-optimal policy. Through maintaining a distribution over scalarization weights, our approach is able to interactively tune a deep RL agent towards a variety of preferences, while eliminating the need for computing multiple policies. We empirically demonstrate the effectiveness of MORAL in two scenarios, which model a delivery and an emergency task that require an agent to act in the presence of normative conflicts. Overall, we consider our research a step towards multi-objective RL with learned rewards, bridging the gap between current reward learning and machine ethics literature.

show abstract

Challenges of Real-World Reinforcement Learning

Cited by 114 publications

References 26 publications

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation

MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning

Contact Info

Product

Resources

About