Feedback Control For Cassie With Deep Reinforcement Learning

Xie, Zhiqiang; Berseth, Glen; Clary, Patrick; Hurst, Jonathan; Panne, Michiel van de

doi:10.1109/iros.2018.8593722

Cited by 170 publications

(136 citation statements)

References 13 publications

Supporting

Mentioning

136

Contrasting

Order By: Relevance

“…Deep RL has been used extensively to learn locomotion policies in simulation [4,20,34,50] and even transfer them to real-world robots [24,44], but this inevitably incurs a loss of performance due to discrepancies in the simulation, and requires accurate system identification. Using such algorithms directly in the real world has proven challenging.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Learning to Walk Via Deep Reinforcement Learning

Haarnoja¹,

Ha²,

Zhou³

et al. 2019

Robotics: Science and Systems XV

302

237

View full text Add to dashboard Cite

Deep reinforcement learning (deep RL) holds the promise of automating the acquisition of complex controllers that can map sensory inputs directly to low-level actions. In the domain of robotic locomotion, deep RL could enable learning locomotion skills with minimal engineering and without an explicit model of the robot dynamics. Unfortunately, applying deep RL to real-world robotic tasks is exceptionally difficult, primarily due to poor sample complexity and sensitivity to hyperparameters. While hyperparameters can be easily tuned in simulated domains, tuning may be prohibitively expensive on physical systems, such as legged robots, that can be damaged through extensive trial-and-error learning. In this paper, we propose a sample-efficient deep RL algorithm based on maximum entropy RL that requires minimal per-task tuning and only a modest number of trials to learn neural network policies. We apply this method to learning walking gaits on a real-world Minitaur robot. Our method can acquire a stable gait from scratch directly in the real world in about two hours, without relying on any model or simulation, and the resulting policy is robust to moderate variations in the environment. We further show that our algorithm achieves state-of-the-art performance on simulated benchmarks with a single set of hyperparameters. Videos of training and the learned policy can be found on the project website 3 .

show abstract

Section: Related Workmentioning

confidence: 99%

“…The policy is trained only on a flat terrain, but the learned gait is robust and can handle obstacles that were not seen during training. learning of locomotion gaits in simulation [4,20,34,50], requiring accurate system identification and modeling.…”

Section: Introductionmentioning

confidence: 99%

Learning to Walk Via Deep Reinforcement Learning

Haarnoja¹,

Ha²,

Zhou³

et al. 2019

Robotics: Science and Systems XV

302

237

View full text Add to dashboard Cite

show abstract

“…Schulman et al [22] trained a locomotion policy for a similar 2D walker with an actor-critic method. More recent work obtained full 3D locomotion policies [23][24][25][26]. In these papers, animated characters achieve remarkable motor skills in simulation.…”

Section: Introductionmentioning

confidence: 99%

Learning agile and dynamic motor skills for legged robots

et al. 2019

View full text Add to dashboard Cite

Legged robots pose one of the greatest challenges in robotics. Dynamic and agile maneuvers of animals cannot be imitated by existing methods that are crafted by humans. A compelling alternative is reinforcement learning, which requires minimal craftsmanship and promotes the natural evolution of a control policy. However, so far, reinforcement learning research for legged robots is mainly limited to simulation, and only few and comparably simple examples have been deployed on real systems. The primary reason is that training with real robots, particularly with dynamically balancing systems, is complicated and expensive. In the present work, we report a new method for training a neural network policy in simulation and transferring it to a state-of-the-art legged system, thereby we leverage fast, automated, and cost-effective data generation schemes. The approach is applied to the ANYmal robot, a sophisticated medium-dog-sized quadrupedal system. Using policies trained in simulation, the quadrupedal machine achieves locomotion skills that go beyond what had been achieved with prior methods: ANYmal is capable of precisely and energy-efficiently following high-level body velocity commands, running faster than ever before, and recovering from falling even in complex configurations.

show abstract

“…A variety of approaches, starting from inverted pendulum models [2], zero moment points [3], passivity based control [4], [5], capture points [6], hybrid zero dynamics [7] to even machine learning based approaches [8], [9] have been explored. A more recent trend has been the use of deep reinforcement learning (D-RL) methods [10], [11] to determine optimal policies for efficient and robust walking behaviors in both bipeds and quadrupeds. D-RL was successfully used to achieve walking in the quadrupedal robot Minitaur [10], where the control policy was trained via one of the well known policy gradient based approaches called the proximal policy optimization (PPO) [12].…”

Section: Introductionmentioning

confidence: 99%

Realizing Learned Quadruped Locomotion Behaviors through Kinematic Motion Primitives

Singla

Bhattacharya

Dholakiya

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Humans and animals are believed to use a very minimal set of trajectories to perform a wide variety of tasks including walking. Our main objective in this paper is two fold 1) Obtain an effective tool to realize these basic motion patterns for quadrupedal walking, called the kinematic motion primitives (kMPs), via trajectories learned from deep reinforcement learning (D-RL) and 2) Realize a set of behaviors, namely trot, walk, gallop and bound from these kinematic motion primitives in our custom four legged robot, called the "Stoch". D-RL is a data driven approach, which has been shown to be very effective for realizing all kinds of robust locomotion behaviors, both in simulation and in experiment. On the other hand, kMPs are known to capture the underlying structure of walking and yield a set of derived behaviors. We first generate walking gaits from D-RL, which uses policy gradient based approaches. We then analyze the resulting walking by using principal component analysis. We observe that the kMPs extracted from PCA followed a similar pattern irrespective of the type of gaits generated. Leveraging on this underlying structure, we then realize walking in Stoch by a straightforward reconstruction of joint trajectories from kMPs. This type of methodology improves the transferability of these gaits to real hardware, lowers the computational overhead on-board, and also avoids multiple training iterations by generating a set of derived behaviors from a single learned gait.

show abstract

Feedback Control For Cassie With Deep Reinforcement Learning

Cited by 170 publications

References 13 publications

Learning to Walk Via Deep Reinforcement Learning

Learning to Walk Via Deep Reinforcement Learning

Learning agile and dynamic motor skills for legged robots

Realizing Learned Quadruped Locomotion Behaviors through Kinematic Motion Primitives

Contact Info

Product

Resources

About