Preparing for the Unknown: Learning a Universal Policy with Online System Identification

Yu, Wenhao; Tan, Jie; Liu, C. Karen; Turk, Greg

doi:10.48550/arxiv.1702.02453

Cited by 52 publications

(75 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The idea of MEP shares a common intuition with domain randomization, where some features of the environment are changed randomly during training to make the policy robust to that feature [45,50,34,42,1,43]. MEP can be seen as a domain randomization technique, where the randomization is conducted over a set of partners' policies.…”

Section: Related Workmentioning

confidence: 99%

Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

Zhao¹,

Song²,

Hu³

et al. 2021

Preprint

View full text Add to dashboard Cite

An AI agent should be able to coordinate with humans to solve tasks. We consider the problem of training a Reinforcement Learning (RL) agent without using any human data, i.e., in a zero-shot setting, to make it capable of collaborating with humans. Standard RL agents learn through self-play. Unfortunately, these agents only know how to collaborate with themselves and normally do not perform well with unseen partners, such as humans. The methodology of how to train a robust agent in a zero-shot fashion is still subject to research. Motivated from the maximum entropy RL, we derive a centralized population entropy objective to facilitate learning of a diverse population of agents, which is later used to train a robust agent to collaborate with unseen partners. The proposed method shows its effectiveness compared to baseline methods, including self-play PPO, the standard Population-Based Training (PBT), and trajectory diversity-based PBT, in the popular Overcooked game environment. We also conduct online experiments with real humans and further demonstrate the efficacy of the method in the real world. A supplementary video showing experimental results is available at https://youtu.be/Xh-FKD0AAKE.

show abstract

Section: Related Workmentioning

confidence: 99%

Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

Zhao¹,

Song²,

Hu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…This type of problem can also be solved by learning a hierarchical policy (Kupcsik et al, 2013), where the upper layer depends on the context and the lower one on the current state of the robot. Similarly (Yu et al, 2017) learns a unique policy in a several environment and use an identification method that feeds a context signal to the policy. The drawback of these methods is that they require access to a wide range of environment in order to learn a latent representation of the dynamics.…”

Section: Related Workmentioning

confidence: 99%

Learning Robust Controllers Via Probabilistic Model-Based Policy Search

Charvet¹,

Jensen²,

Murray-Smith³

2021

Preprint

View full text Add to dashboard Cite

Model-based Reinforcement Learning estimates the true environment through a world model in order to approximate the optimal policy. This family of algorithms usually benefits from better sample efficiency than their model-free counterparts. We investigate whether controllers learned in such a way are robust and able to generalize under small perturbations of the environment. Our work is inspired by the PILCO algorithm, a method for probabilistic policy search. We show that enforcing a lower bound to the likelihood noise in the Gaussian Process dynamics model regularizes the policy updates and yields more robust controllers. We demonstrate the empirical benefits of our method in a simulation benchmark.

show abstract

“…Robot learning [1]- [3] has been testified to successfully work not only in simulation but also on real-world robotic control. In this domain, reinforcement learning (RL) [4]- [6] methods are typically applied for robotic control via sim-toreal transfer [2], [7], [8].…”

Section: Introductionmentioning

confidence: 99%

“…SI methods usually configure the physical parameters from historical transition data, either in an explicit or implicit manner. Both approaches have been proven to be feasible for some control tasks [1], [3], when encountering the sim-to-real transfer problems, or more generally, domain transfer problems. However, a successful execution of the task does not necessarily indicate that an optimal control strategy is achieved, which leaves the space for further improvement based on solving the existing defects of above approaches.…”

Section: Introductionmentioning

confidence: 99%

“…SI can help with bridging this gap via an active inferring of system dynamics with the necessary data collected. In [1], an online system identification (OSI) module is used for explicitly configuring the dynamics parameters of environments, with a subsequent universal policy (UP) conditioned on the identified parameters to generate adaptive actions in new environments. However, both the UP and OSI modules can be infeasible in practice for general robot learning tasks according to the discussions in [8].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Not Only Domain Randomization: Universal Policy with Embedding System Identification

Ding¹

2021

Preprint

View full text Add to dashboard Cite

Domain randomization (DR) cannot provide optimal policies for adapting the learning agent to the dynamics of the environment, although it can generalize sub-optimal policies to work in a transferred domain. In this paper, we present Universal Policy with Embedding System Identification (UPESI) as an implicit system identification (SI) approach with universal policies (UPs), as a learning-based control method to execute optimal actions adaptively in environments with various dynamic properties. Previous approaches of SI for adaptive policies either conduct explicit SI, which is testified to be an ill-posed problem, or suffer from low efficiency without leveraging the simulation oracle. We propose to conduct SI in the embedding space of system dynamics by leveraging a learned forward dynamics model, and use Bayesian optimization for the SI process given transition data in a new environment. The identified embeddings are applied as additional input to the UP to enable its dynamics adaptability. Experiments demonstrate the advantageous performances of our proposed UP with embedding SI over standard DR and conventional SI approaches on both low-dimensional and high-dimensional simulation tasks.

show abstract

Preparing for the Unknown: Learning a Universal Policy with Online System Identification

Cited by 52 publications

References 9 publications

Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

Learning Robust Controllers Via Probabilistic Model-Based Policy Search

Not Only Domain Randomization: Universal Policy with Embedding System Identification

Contact Info

Product

Resources

About