Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Composition

Siekmann, Jonah; Godse, Yesh; Fern, Alan; Hurst, Jonathan

doi:10.1109/icra48506.2021.9561814

Cited by 101 publications

(70 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These complex behaviors can be composed of a single framework using pre-trained expert networks and a gate neural network, and manifest agile and effective motions [7] on the real robot. Furthermore, RL can be utilized for bipedal robots to climb up stairs [8] or to display diverse locomotion patterns such as standing, walking, and running [9].…”

Section: Introductionmentioning

confidence: 99%

Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion

Mun

Kim

et al. 2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

In this paper, we propose a locomotion training framework where a control policy and a state estimator are trained concurrently. The framework consists of a policy network which outputs the desired joint positions and a state estimation network which outputs estimates of the robot's states such as the base linear velocity, foot height, and contact probability. We exploit a fast simulation environment to train the networks and the trained networks are transferred to the real robot. The trained policy and state estimator are capable of traversing diverse terrains such as a hill, slippery plate, and bumpy road. We also demonstrate that the learned policy can run at up to 3.75 m/s on normal flat ground and 3.54 m/s on a slippery plate with the coefficient of friction of 0.22.

show abstract

Section: Introductionmentioning

confidence: 99%

Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion

Mun

Kim

et al. 2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

show abstract

“…Reinforcement learning (RL) based approaches have begun to show significant promise at robust real-world legged locomotion [1,12,13]. Unlike optimization or heuristic-based control methods which rely on prescribed ground contact schedules or force-based event detection, RL can produce control policies which learn proprioceptive reflexes and strategies for dealing with unexpectedly early or late contact and rough terrain through exposure to a variety of disturbances during training.…”

Section: Learned Controllermentioning

confidence: 99%

“…In the remainder of this section, we detail the specific simto-real RL formulation used in this work, which follows recent work [13] on learning different biped gaits over flat terrain. Surprisingly, only minimal changes were required to enable policy learning for the much more complex stair-like terrains of this paper 1 In particular, the only major modification required was the randomized domain generation of stair-like rather than mostly flat terrain as discussed later in Section III; no novel stair-specific reward terms were needed.…”

Section: Reinforcement Learning Formulationmentioning

confidence: 99%

“…We use the method introduced in [13] to specify our reward function. To briefly review this method, we desire a reward framework which allows for penalizing the policy for large magnitudes of some quantities of the environment at certain times, while permitting those quantities to be large at other times.…”

Section: Reward Functionmentioning

confidence: 99%

“…For a detailed explanation of the reward function used, see the Appendix. As in [13], we do not rely on expert reference trajectories to learn behaviors.…”

Section: Reward Functionmentioning

confidence: 99%

See 2 more Smart Citations

Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning

Siekmann¹,

Green²,

Warila³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Accurate and precise terrain estimation is a difficult problem for robot locomotion in real-world environments. Thus, it is useful to have systems that do not depend on accurate estimation to the point of fragility. In this paper, we explore the limits of such an approach by investigating the problem of traversing stair-like terrain without any external perception or terrain models on a bipedal robot. For such blind bipedal platforms, the problem appears difficult (even for humans) due to the surprise elevation changes. Our main contribution is to show that sim-to-real reinforcement learning (RL) can achieve robust locomotion over stair-like terrain on the bipedal robot Cassie using only proprioceptive feedback. Importantly, this only requires modifying an existing flat-terrain training RL framework to include stair-like terrain randomization, without any changes in reward function. To our knowledge, this is the first controller for a bipedal, human-scale robot capable of reliably traversing a variety of real-world stairs and other stair-like disturbances using only proprioception.

show abstract