Safety-Aware Apprenticeship Learning

Zhou, Weichao; Li, Wenchao

doi:10.1007/978-3-319-96145-3_38

Cited by 26 publications

(16 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Valko et al (2013) extend the max margin formulation to the semi-supervised case where trajectories provided by an expert are assumed to be the labeled examples and trajectories provided by a non-expert are assumed to be the unlabeled examples. Zhou and Li (2018) develop a safety-aware version of max margin apprenticeship learning. A policy learned through apprenticeship learning can lead an agent into an unsafe state because an expert only supplies positive examples.…”

Section: Maximum Margin Methodsmentioning

confidence: 99%

A survey of inverse reinforcement learning

2022

View full text Add to dashboard Cite

Learning from demonstration, or imitation learning, is the process of learning to act in an environment from examples provided by a teacher. Inverse reinforcement learning (IRL) is a specific form of learning from demonstration that attempts to estimate the reward function of a Markov decision process from examples provided by the teacher. The reward function is often considered the most succinct description of a task. In simple applications, the reward function may be known or easily derived from properties of the system and hard coded into the learning process. However, in complex applications, this may not be possible, and it may be easier to learn the reward function by observing the actions of the teacher. This paper provides a comprehensive survey of the literature on IRL. This survey outlines the differences between IRL and two similar methods - apprenticeship learning and inverse optimal control. Further, this survey organizes the IRL literature based on the principal method, describes applications of IRL algorithms, and provides areas of future research.

show abstract

Section: Maximum Margin Methodsmentioning

confidence: 99%

A survey of inverse reinforcement learning

2022

View full text Add to dashboard Cite

show abstract

“…Using their taxonomy, shielding is an instance of "teacher provides advice" [9], where a teacher with additional information about the system guides the RL agent to pick the right actions. Apprenticeship learning [1] is a closely related variant where the teacher gives (positive) examples and has been used in the context of verification [42]. Uppaal Stratego synthesizes safe, permissive policies that are optimized via learning to create controllers for real-time systems [10].…”

Section: Related Workmentioning

confidence: 99%

Online Shielding for Stochastic Systems

Könighofer

Rudolf

Palmisano

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We propose a method to develop trustworthy reinforcement learning systems. To ensure safety especially during exploration, we automatically synthesize a correct-by-construction runtime enforcer, called a shield, that blocks all actions of the agent that are unsafe with respect to a temporal logic specification. Our main contribution is a new synthesis algorithm for computing the shield online. Existing offline shielding approaches compute exhaustively the safety of all states-action combinations ahead-of-time, resulting in huge computation times, large memory consumption, and significant delays at runtime due to the look-ups in huge databases. The intuition behind online shielding is to compute at runtime the set of all states that could be reached in the near future. For each of these states, the safety of all available actions is analysed and used for shielding as soon as one of the considered states is reached. Our proposed method is general and can be applied to a wide range of planning problems with stochastic behaviour. For our evaluation, we selected a 2player version of the classical computer game Snake. The game requires fast decisions and the multiplayer setting induces a large state space, computationally expensive to analyze exhaustively. The safety objective of collision avoidance is easily transferable to a variety of planning tasks.

show abstract

“…Regarding LfD, Inverse reward design (IRD) [19] design reward functions in a manner similar to IRL. Safety-aware apprenticeship learning [44] incorporate formal specification and formal verification with IRL. However, those works confine the reward function to be linear of features as IRL does.…”

Section: Related Workmentioning

confidence: 99%

Programmatic Reward Design by Example

Zhou¹,

Li²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Reward design is a fundamental problem in reinforcement learning (RL). A misspecified or poorly designed reward can result in low sample efficiency and undesired behaviors. In this paper, we propose the idea of programmatic reward design, i.e. using programs to specify the reward functions in RL environments. Programs allow human engineers to express sub-goals and complex task scenarios in a structured and interpretable way. The challenge of programmatic reward design, however, is that while humans can provide the high-level structures, properly setting the low-level details, such as the right amount of reward for a specific sub-task, remains difficult. A major contribution of this paper is a probabilistic framework that can infer the best candidate programmatic reward function from expert demonstrations. Inspired by recent generative-adversarial approaches, our framework searches for the most likely programmatic reward function under which the optimally generated trajectories cannot be differentiated from the demonstrated trajectories. Experimental results show that programmatic reward functions learned using this framework can significantly outperform those learned using existing reward learning algorithms, and enable RL agents to achieve state-of-the-art performance on highly complex tasks.

show abstract

Safety-Aware Apprenticeship Learning

Cited by 26 publications

References 23 publications

A survey of inverse reinforcement learning

A survey of inverse reinforcement learning

Online Shielding for Stochastic Systems

Programmatic Reward Design by Example

Contact Info

Product

Resources

About