Learning to Walk via Deep Reinforcement Learning

Haarnoja, Tuomas; Ha, Sehoon; Zhou, Aurick; Tan, Jie; Tucker, George; Levine, Sergey

doi:10.48550/arxiv.1812.11103

Cited by 91 publications

(116 citation statements)

References 0 publications

Supporting

Mentioning

116

Contrasting

Order By: Relevance

“…Every module is fed all environment information x k = (x, k) and distinctly chosen IGFs mask which part of the input each network has access to, thereby influencing which skills they learn. By presenting multiple priors, we enable a comparison with existing literature [14,19,20,21]. With the right masking, one can recover previously investigated asymmetries [14,19], explore additional ones, and also express purely hierarchical [9] and KL-regularized equivalents [8].…”

Section: Information Asymmetrymentioning

confidence: 99%

Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning

Salter¹,

Hartikainen²,

Goodwin³

et al. 2022

Preprint

View full text Add to dashboard Cite

The ability to discover behaviours from past experience and transfer them to new tasks is a hallmark of intelligent agents acting sample-efficiently in the real world. Equipping embodied reinforcement learners with the same ability may be crucial for their successful deployment in robotics. While hierarchical and KL-regularized RL individually hold promise here, arguably a hybrid approach could combine their respective benefits. Key to these fields is the use of information asymmetry to bias which skills are learnt. While asymmetric choice has a large influence on transferability, prior works have explored a narrow range of asymmetries, primarily motivated by intuition. In this paper, we theoretically and empirically show the crucial trade-off, controlled by information asymmetry, between the expressivity and transferability of skills across sequential tasks. Given this insight, we provide a principled approach towards choosing asymmetry and apply our approach to a complex, robotic block stacking domain, unsolvable by baselines, demonstrating the effectiveness of hierarchical KL-regularized RL, coupled with correct asymmetric choice, for sample-efficient transfer learning.

show abstract

Section: Information Asymmetrymentioning

confidence: 99%

Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning

Salter¹,

Hartikainen²,

Goodwin³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Learning for Legged Locomotion and Gaits: Data-driven learning for legged locomotion has shown robust controllers for quadrupedal robots [22,34,35,36,37]. Of these, [34,22] focus on a controller on complex terrains but do not synthesize and analyze leg patterns of the robot across diverse gaits at different target speeds.…”

Section: Related Workmentioning

confidence: 99%

Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots

Fu¹,

Kumar²,

Malik³

et al. 2021

Preprint

View full text Add to dashboard Cite

Legged locomotion is commonly studied and expressed as a discrete set of gait patterns, like walk, trot, gallop, which are usually treated as given and pre-programmed in legged robots for efficient locomotion at different speeds. However, fixing a set of pre-programmed gaits limits the generality of locomotion. Recent animal motor studies show that these conventional gaits are only prevalent in ideal flat terrain conditions while real-world locomotion is unstructured and more like bouts of intermittent steps. What principles could lead to both structured and unstructured patterns across mammals and how to synthesize them in robots? In this work, we take an analysis-by-synthesis approach and learn to move by minimizing mechanical energy. We demonstrate that learning to minimize energy consumption plays a key role in the emergence of natural locomotion gaits at different speeds in real quadruped robots. The emergent gaits are structured in ideal terrains and look similar to that of horses and sheep. The same approach leads to unstructured gaits in rough terrains which is consistent with the findings in animal motor control. We validate our hypothesis in both simulation and real hardware across natural terrains. Videos at https://energy-locomotion.github.io.

show abstract

“…We also validate the policy with successful transfer to hardware. Previously, [22] trained policies in Minitaur quadruped in 160 epochs using SAC. Due to increasing complexity, we will consider direct hardware training for quadrupeds in future.…”

Section: Hardware Training and Transfermentioning

confidence: 99%

Dynamic Mirror Descent based Model Predictive Control for Accelerating Robot Learning

Mishra¹,

Samineni²,

Goel³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent works in Reinforcement Learning (RL) combine model-free (Mf)-RL algorithms with model-based (Mb)-RL approaches to get the best from both: asymptotic performance of Mf-RL and high sample-efficiency of Mb-RL.Inspired by these works, we propose a hierarchical framework that integrates online learning for the Mb-trajectory optimization with off-policy methods for the Mf-RL. In particular, two loops are proposed, where the Dynamic Mirror Descent based Model Predictive Control (DMD-MPC) is used as the inner loop Mb-RL to obtain an optimal sequence of actions. These actions are in turn used to significantly accelerate the outer loop Mf-RL. We show that our formulation is generic for a broad class of MPC based policies and objectives, and includes some of the well-known Mb-Mf approaches. We finally introduce a new algorithm: Mirror-Descent Model Predictive RL (M-DeMoRL), which uses Cross-Entropy Method (CEM) with elite fractions for the inner loop. Our experiments show faster convergence of the proposed hierarchical approach on benchmark MuJoCo tasks. We also demonstrate hardware training for trajectory tracking in a 2R leg, and hardware transfer for robust walking in a quadruped. We show that the inner-loop Mb-RL significantly decreases the number of training iterations required in the real system, thereby validating the proposed approach.

show abstract

Learning to Walk via Deep Reinforcement Learning

Cited by 91 publications

References 0 publications

Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning

Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning

Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots

Dynamic Mirror Descent based Model Predictive Control for Accelerating Robot Learning

Contact Info

Product

Resources

About