Model-free Reinforcement Learning for Robust Locomotion using Demonstrations from Trajectory Optimization

Bogdanovic, Miroslav; Khadiv, Majid; Righetti, Ludovic

doi:10.48550/arxiv.2107.06629

Cited by 6 publications

(10 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Model-free RL for Robust Locomotion using Demonstrations from Trajectory Optimization [109] ArXiv 2022…”

Section: Isaac Pybullt Unitree A1mentioning

confidence: 99%

See 1 more Smart Citation

Deep reinforcement learning for real-world quadrupedal locomotion: a comprehensive review

Zhang¹,

Li²,

Wang³

2022

Intell Robot

View full text Add to dashboard Cite

Building controllers for legged robots with agility and intelligence has been one of the typical challenges in the pursuit of artificial intelligence (AI). As an important part of the AI field, deep reinforcement learning (DRL) can realize sequential decision making without physical modeling through end-to-end learning and has achieved a series of major breakthroughs in quadrupedal locomotion research. In this review article, we systematically organize and summarize relevant important literature, covering DRL algorithms from problem setting to advanced learning methods. These algorithms alleviate the specific problems encountered in the practical application of robots to a certain extent. We first elaborate on the general development trend in this field from several aspects, such as the DRL algorithms, simulation environments, and hardware platforms. Moreover, core components in the algorithm design, such as state and action spaces, reward functions, and solutions to reality gap problems, are highlighted and summarized. We further discuss open problems and propose promising future research directions to discover new areas of research.

show abstract

“…Model-free RL for Robust Locomotion using Demonstrations from Trajectory Optimization [109] ArXiv 2022…”

Section: Isaac Pybullt Unitree A1mentioning

confidence: 99%

“…Model-free Reinforcement Learning for Robust Locomotion using Demonstrations from Trajectory Optimization [109] Trajectory Optimization Algorithm [118] .…”

Section: Financial Support and Sponsorshipmentioning

confidence: 99%

Deep reinforcement learning for real-world quadrupedal locomotion: a comprehensive review

Zhang¹,

Li²,

Wang³

2022

Intell Robot

View full text Add to dashboard Cite

show abstract

“…To provide the user more control over the behaviors, reference trajectories can be provided to encourage desired motions. One can design a reward function to explicitly track the reference trajectories, e.g., [15], [16], [17]. Inverse reinforcement learning techniques such as adversarial motion priors can also be used to learn a reward function to encourage the policy to produce motions that look similar to a prescribed motion dataset, e.g., [18], [19].…”

Section: Imitation-based Reinforcement Learning For Legged Robotsmentioning

confidence: 99%

OPT-Mimic: Imitation of Optimized Trajectories for Dynamic Quadruped Behaviors

Fuchioka¹,

Xie²,

Panne³

2022

Preprint

View full text Add to dashboard Cite

Reinforcement Learning (RL) has seen many recent successes for quadruped robot control. The imitation of reference motions provides a simple and powerful prior for guiding solutions towards desired solutions without the need for meticulous reward design. While much work uses motion capture data or hand-crafted trajectories as the reference motion, relatively little work has explored the use of reference motions coming from model-based trajectory optimization. In this work, we investigate several design considerations that arise with such a framework, as demonstrated through four dynamic behaviours: trot, front hop, 180 backflip, and biped stepping. These are trained in simulation and transferred to a physical Solo 8 quadruped robot without further adaptation. In particular, we explore the space of feed-forward designs afforded by the trajectory optimizer to understand its impact on RL learning efficiency and sim-to-real transfer. These findings contribute to the long standing goal of producing robot controllers that combine the interpretability and precision of model-based optimization with the robustness that model-free RL-based controllers offer.

show abstract

“…In very recent work [22] that is perhaps most similar to ours, a single trajectory was generated with a motion planner and RL was used to imitate it in a timing independent way and learn a bounding or hopping behavior which was finetuned and evaluated on a robot. Our work is different in that we generate an entire dataset of trajectories and use it for learning terrain adaptive policies that can generalize and be fine-tuned for a variety of challenging terrains.…”

Section: Planner Imitation For Legged Robotsmentioning

confidence: 99%

Learning Coordinated Terrain-Adaptive Locomotion by Imitating a Centroidal Dynamics Planner

Brakel¹,

Bohez²,

Hasenclever³

et al. 2021

Preprint

View full text Add to dashboard Cite

Dynamic quadruped locomotion over challenging terrains with precise foot placements is a hard problem for both optimal control methods and Reinforcement Learning (RL). Non-linear solvers can produce coordinated constraint satisfying motions, but often take too long to converge for online application. RL methods can learn dynamic reactive controllers but require carefully tuned shaping rewards to produce good gaits and can have trouble discovering precise coordinated movements. Imitation learning circumvents this problem and has been used with motion capture data to extract quadruped gaits for flat terrains. However, it would be costly to acquire motion capture data for a very large variety of terrains with height differences. In this work, we combine the advantages of trajectory optimization and learning methods and show that terrain adaptive controllers can be obtained by training policies to imitate trajectories that have been planned over procedural terrains by a non-linear solver. We show that the learned policies transfer to unseen terrains and can be fine-tuned to dynamically traverse challenging terrains that require precise foot placements and are very hard to solve with standard RL.

show abstract

Model-free Reinforcement Learning for Robust Locomotion using Demonstrations from Trajectory Optimization

Cited by 6 publications

References 13 publications

Deep reinforcement learning for real-world quadrupedal locomotion: a comprehensive review

Deep reinforcement learning for real-world quadrupedal locomotion: a comprehensive review

OPT-Mimic: Imitation of Optimized Trajectories for Dynamic Quadruped Behaviors

Learning Coordinated Terrain-Adaptive Locomotion by Imitating a Centroidal Dynamics Planner

Contact Info

Product

Resources

About