Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning

Siekmann, Jonah; Green, Kevin; Warila, John; Fern, Alan; Hurst, Jonathan

doi:10.48550/arxiv.2105.08328

Cited by 15 publications

(26 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2) uses the same graph morphologies as Multi-Ped, but instead of learning locomotion strategies on flat terrain, the agent has to navigate a stairs-like pattern of blocks. The state space does not contain any information about the current positions or heights of blocks making the agent navigate the terrain blindly (similarly to [46]). The rewards and design parameters are the same as the Multi-Ped environment.…”

Section: Methodsmentioning

confidence: 99%

What Robot do I Need? Fast Co-Adaptation of Morphology and Control using Graph Neural Networks

Luck¹,

Calandra²,

Mistry³

2021

Preprint

View full text Add to dashboard Cite

The co-adaptation of robot morphology and behaviour becomes increasingly important with the advent of fast 3D-manufacturing methods and efficient deep reinforcement learning algorithms. A major challenge for the application of coadaptation methods to the real world is the simulation-to-reality-gap due to model and simulation inaccuracies. However, prior work focuses primarily on the study of evolutionary adaptation of morphologies exploiting analytical models and (differentiable) simulators with large population sizes, neglecting the existence of the simulation-to-reality-gap and the cost of manufacturing cycles in the real world. This paper presents a new approach combining classic high-frequency deep neural networks with computational expensive Graph Neural Networks for the data-efficient co-adaptation of agents with varying numbers of degrees-of-freedom. Evaluations in simulation show that the new method can co-adapt agents within such a limited number of production cycles by efficiently combining design optimization with offline reinforcement learning, that it allows for the direct application to real-world co-adaptation tasks in future work.Preprint. Under review.

show abstract

Section: Methodsmentioning

confidence: 99%

What Robot do I Need? Fast Co-Adaptation of Morphology and Control using Graph Neural Networks

Luck¹,

Calandra²,

Mistry³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…[10,11,12]), the majority of works with real robots have focused on approaches that learn locomotion skills in simulation, and then transfer the resulting con-trollers to the hardware [13,14,15,16]. Examples include the traversal of rough terrain with a quadruped [17,18] and robust walking and stair climbing with a biped [19,20,21]. With a sufficiently accurate simulation model (for instance via learned actuator models [13]) and policies that are robust to or can adapt to distribution shift (e.g.…”

Section: Introductionmentioning

confidence: 99%

Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviors

Bohez¹,

Tunyasuvunakool²,

Brakel³

et al. 2022

Preprint

View full text Add to dashboard Cite

We investigate the use of prior knowledge of human and animal movement to learn reusable locomotion skills for real legged robots. Our approach builds upon previous work on imitating human or dog Motion Capture (MoCap) data to learn a movement skill module. Once learned, this skill module can be reused for complex downstream tasks. Importantly, due to the prior imposed by the MoCap data, our approach does not require extensive reward engineering to produce sensible and natural looking behavior at the time of reuse. This makes it easy to create well-regularized, task-oriented controllers that are suitable for deployment on real robots. We demonstrate how our skill module can be used for imitation, and train controllable walking and ball dribbling policies for both the ANYmal quadruped and OP3 humanoid. These policies are then deployed on hardware via zero-shot simulation-to-reality transfer. Accompanying videos are available at https://bit.ly/robot-npmp.

show abstract

“…While recent hardware and algorithmic developments have led to impressive legged locomotion demonstrations of both learned controllers [1], [2], model-based planning approaches [3], [4] and combinations of the two [5], [6], there appears to be a trade-off between the overall agility of the systems and the ability to traverse complex terrains that require precise foot placements. Broadly speaking, methods that involve planning tend to struggle more with dynamic movements while learned controllers struggle more with producing precise coordinated foot placements and desirable gaits.…”

Section: Introductionmentioning

confidence: 99%

“…Reinforcement learning (RL) methods generate closedloop control policies which can produce very dynamic and robust gaits that have been successfully transferred to hardware [2], [7]. However, it often requires significant tuning and the use of many shaping rewards to produce desirable gaits.…”

Section: Introductionmentioning

confidence: 99%

Learning Coordinated Terrain-Adaptive Locomotion by Imitating a Centroidal Dynamics Planner

Brakel¹,

Bohez²,

Hasenclever³

et al. 2021

Preprint

View full text Add to dashboard Cite

Dynamic quadruped locomotion over challenging terrains with precise foot placements is a hard problem for both optimal control methods and Reinforcement Learning (RL). Non-linear solvers can produce coordinated constraint satisfying motions, but often take too long to converge for online application. RL methods can learn dynamic reactive controllers but require carefully tuned shaping rewards to produce good gaits and can have trouble discovering precise coordinated movements. Imitation learning circumvents this problem and has been used with motion capture data to extract quadruped gaits for flat terrains. However, it would be costly to acquire motion capture data for a very large variety of terrains with height differences. In this work, we combine the advantages of trajectory optimization and learning methods and show that terrain adaptive controllers can be obtained by training policies to imitate trajectories that have been planned over procedural terrains by a non-linear solver. We show that the learned policies transfer to unseen terrains and can be fine-tuned to dynamically traverse challenging terrains that require precise foot placements and are very hard to solve with standard RL.

show abstract

Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning

Cited by 15 publications

References 7 publications

What Robot do I Need? Fast Co-Adaptation of Morphology and Control using Graph Neural Networks

What Robot do I Need? Fast Co-Adaptation of Morphology and Control using Graph Neural Networks

Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviors

Learning Coordinated Terrain-Adaptive Locomotion by Imitating a Centroidal Dynamics Planner

Contact Info

Product

Resources

About