Robust Feedback Motion Policy Design Using Reinforcement Learning on a 3D Digit Bipedal Robot

Castillo, Guillermo A.; Weng, Bowen; Zhang, Wei; Hereid, Ayonga

doi:10.1109/iros51168.2021.9636467

Cited by 53 publications

(27 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It can even prevent learning anything if the variability is not increased progressively [10]. Alternatively, the stability can be improved by predicting high-level features for a modelbased controller [32], but it bounds the overall performance. Besides, a memory network or a history of previous timesteps is often used to thwart partial observability of the state [10], [31].…”

Section: Simulation To Real World Transfermentioning

confidence: 99%

Reactive Stepping for Humanoid Robots using Reinforcement Learning: Application to Standing Push Recovery on the Exoskeleton Atalante

Duburcq¹,

Schramm²,

Boéris³

et al. 2022

Preprint

View full text Add to dashboard Cite

State-of-the-art reinforcement learning is now able to learn versatile locomotion, balancing and push-recovery capabilities for bipedal robots in simulation. Yet, the reality gap has mostly been overlooked and the simulated results hardly transfer to real hardware. Either it is unsuccessful in practice because the physics is over-simplified and hardware limitations are ignored, or regularity is not guaranteed and unexpected hazardous motions can occur. This paper presents a reinforcement learning framework capable of learning robust standing push recovery for bipedal robots with a smooth out-of-the-box transfer to reality, requiring only instantaneous proprioceptive observations. By combining original termination conditions and policy smoothness conditioning, we achieve stable learning, simto-real transfer and safety using a policy without memory nor observation history. Reward shaping is then used to give insights into how to keep balance. We demonstrate its performance in reality on the lower-limb medical exoskeleton Atalante.

show abstract

Section: Simulation To Real World Transfermentioning

confidence: 99%

Reactive Stepping for Humanoid Robots using Reinforcement Learning: Application to Standing Push Recovery on the Exoskeleton Atalante

Duburcq¹,

Schramm²,

Boéris³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The gait controller is responsible for keeping track of the gait parameters and tracking the generated foot trajectory. Based on the contact state of the leg (estimated as explained in Section II-A.3), the gait controller augments the ankle regulation followed by joint level PD tracking for the swing leg and just the torso regulation for the stance leg, as in [13]. A phase-variable τ, τ ∈ [0, 1) which is used to track the semi-elliptical trajectory gets reset once every walking step or upon a premature foot contact.…”

Section: Gait Controllermentioning

confidence: 99%

“…The swing foot is kept parallel to the underlying terrain elevation to ensure proper landing on the ground. The desired position of the swing foot is determined from the kinematics of the robot's leg [13] as follows: where q d tr and q d tp are the target angles for the ankle roll and pitch joints. The value of S f is defined as follows, S f = −1 left leg in swing phase +1 right leg in swing phase Torso Regulation: The torso regulation is applied to ensure an upright torso, which is desired for a stable walking gait and, more importantly, to prevent the stance leg from sliding.…”

Section: Gait Controllermentioning

confidence: 99%

“…3) Comparison with Baselines: As the proposed methodology expresses the effectiveness of a simple linear behavioral policy to control Digit, we present comparison with a recent baseline on the robot with a compact non-linear neural network policy [13]. The key difference between the current and baseline algorithm, is planning in task space and joint space of the robot, respectively.…”

Section: ) Performance Analysismentioning

confidence: 99%

“…Deep Reinforcement Learning (DRL) in robotics, and particularly in the context of legged locomotion [7], [8], [9], [10] has shown appreciable progress in the development of state-of-the-art (SOTA) control frameworks. It has witnessed wide applications ranging from simulated physics-based animations [11], [12] to robust movements in real hardware [13], [10]. With the DRL framework, we let the bipeds learn to walk by themselves, thereby avoiding the complex tuning process.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Linear Policies are Sufficient to Realize Robust Bipedal Walking on Challenging Terrains

Krishna¹,

Castillo²,

Mishra³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In this work, we demonstrate robust walking in the bipedal robot Digit on uneven terrains by just learning a single linear policy. In particular, we propose a new control pipeline, wherein the high-level trajectory modulator shapes the end-foot ellipsoidal trajectories, and the low-level gait controller regulates the torso and ankle orientation. The foot-trajectory modulator uses a linear policy and the regulator uses a linear PD control law. As opposed to neural network based policies, the proposed linear policy has only 13 learnable parameters, thereby not only guaranteeing sample efficient learning but also enabling simplicity and interpretability of the policy. This is achieved with no loss of performance on challenging terrains like slopes, stairs and outdoor landscapes. We first demonstrate robust walking in the custom simulation environment, MuJoCo, and then directly transfer to hardware with no modification of the control pipeline. We subject the biped to a series of pushes and terrain height changes, both indoors and outdoors, thereby validating the presented work.

show abstract

Enabling Surrogate-Assisted Evolutionary Reinforcement Learning via Policy Embedding

Tang¹,

Li²,

Zhang³

et al. 2023

Communications in Computer and Information Science

View full text Add to dashboard Cite

Robust Feedback Motion Policy Design Using Reinforcement Learning on a 3D Digit Bipedal Robot

Cited by 53 publications

References 17 publications

Reactive Stepping for Humanoid Robots using Reinforcement Learning: Application to Standing Push Recovery on the Exoskeleton Atalante

Reactive Stepping for Humanoid Robots using Reinforcement Learning: Application to Standing Push Recovery on the Exoskeleton Atalante

Linear Policies are Sufficient to Realize Robust Bipedal Walking on Challenging Terrains

Enabling Surrogate-Assisted Evolutionary Reinforcement Learning via Policy Embedding

Contact Info

Product

Resources

About