Sim-to-Real: Learning Agile Locomotion For Quadruped Robots

Tan, Jie; Zhang, Tingnan; Coumans, Erwin; İşçen, Atıl; Bai, Yuhang; Hafner, Danijar; Bohez, Steven; Vanhoucke, Vincent

doi:10.48550/arxiv.1804.10332

Cited by 122 publications

(179 citation statements)

References 25 publications

Supporting

Mentioning

179

Contrasting

Order By: Relevance

“…The idea of MEP shares a common intuition with domain randomization, where some features of the environment are changed randomly during training to make the policy robust to that feature [45,50,34,42,1,43]. MEP can be seen as a domain randomization technique, where the randomization is conducted over a set of partners' policies.…”

Section: Related Workmentioning

confidence: 99%

Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

Zhao¹,

Song²,

Hu³

et al. 2021

Preprint

View full text Add to dashboard Cite

An AI agent should be able to coordinate with humans to solve tasks. We consider the problem of training a Reinforcement Learning (RL) agent without using any human data, i.e., in a zero-shot setting, to make it capable of collaborating with humans. Standard RL agents learn through self-play. Unfortunately, these agents only know how to collaborate with themselves and normally do not perform well with unseen partners, such as humans. The methodology of how to train a robust agent in a zero-shot fashion is still subject to research. Motivated from the maximum entropy RL, we derive a centralized population entropy objective to facilitate learning of a diverse population of agents, which is later used to train a robust agent to collaborate with unseen partners. The proposed method shows its effectiveness compared to baseline methods, including self-play PPO, the standard Population-Based Training (PBT), and trajectory diversity-based PBT, in the popular Overcooked game environment. We also conduct online experiments with real humans and further demonstrate the efficacy of the method in the real world. A supplementary video showing experimental results is available at https://youtu.be/Xh-FKD0AAKE.

show abstract

Section: Related Workmentioning

confidence: 99%

Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

Zhao¹,

Song²,

Hu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Reinforcement learning has been demonstrated to be a promising tool to build controllers for legged robotics [7]- [10]. With well-designed reward functions, controller for locomotion can be trained from simulation and deployed to hardware [7].…”

Section: B Reference-based Rl Controllers For Legged Roboticsmentioning

confidence: 99%

“…To reproduce the agile locomotion, one key step is to reproduce each gait and all the transitions with a robust controller for quadruped robots. The state-of-the-art controllers, including model-based controllers [3]- [6] and reinforcement learning (RL) controllers [7]- [10], have achieved excellent performance in outdoor locomotion tests with specific gaits. Both types of controllers, however, can hardly make free transitions according to speed or terrains.…”

Section: Introductionmentioning

confidence: 99%

Learning Free Gait Transition for Quadruped Robots Via Phase-Guided Controller

Shao

Jin

Liu

et al. 2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Gaits and transitions are key components in legged locomotion. For legged robots, describing and reproducing gaits as well as transitions remain longstanding challenges. Reinforcement learning has become a powerful tool to formulate controllers for legged robots. Learning multiple gaits and transitions, nevertheless, is related to the multi-task learning problems. In this work, we present a novel framework for training a simple control policy for a quadruped robot to locomote in various gaits. Four independent phases are used as the interface between the gait generator and the control policy, which characterizes the movement of four feet. Guided by the phases, the quadruped robot is able to locomote according to the generated gaits, such as walk, trot, pacing and bounding, and to make transitions among those gaits. More general phases can be used to generate complex gaits, such as mixed rhythmic dancing. With the control policy, the Black Panther robot, a medium-dog-sized quadruped robot, can perform all learned motor skills while following the velocity commands smoothly and robustly in natural environment.

show abstract

“…2) Student: The student policy is distilled from the teacher policy by minimizing the loss function in (9), where L action penalizes the difference between the student and teacher action, and L embedding regresses the student wrench and exteroceptive embeddings towards those from the teacher. L total = L action + L embedding + L decoder (9) L decoder = L privileged + L scan + L w1 + L w2 (10) In (10), L w1 is the scaled mean-squared error between the decoded wrench and the current external wrench applied to the robot base.…”

Section: Base Policy Trainingmentioning

confidence: 99%

Combining Learning-based Locomotion Policy with Model-based Manipulation for Legged Mobile Manipulators

Ma¹,

Farshidian²,

Miki³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep reinforcement learning produces robust locomotion policies for legged robots over challenging terrains. To date, few studies have leveraged model-based methods to combine these locomotion skills with the precise control of manipulators. Here, we incorporate external dynamics plans into learning-based locomotion policies for mobile manipulation. We train the base policy by applying a random wrench sequence on the robot base in simulation and add the noisified wrench sequence prediction to the policy observations. The policy then learns to counteract the partially-known future disturbance. The random wrench sequences are replaced with the wrench prediction generated with the dynamics plans from model predictive control to enable deployment. We show zero-shot adaptation for manipulators unseen during training. On the hardware, we demonstrate stable locomotion of legged robots with the prediction of the external wrench.

show abstract

Sim-to-Real: Learning Agile Locomotion For Quadruped Robots

Cited by 122 publications

References 25 publications

Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

Learning Free Gait Transition for Quadruped Robots Via Phase-Guided Controller

Combining Learning-based Locomotion Policy with Model-based Manipulation for Legged Mobile Manipulators

Contact Info

Product

Resources

About