Real-Time Trajectory Adaptation for Quadrupedal Locomotion using Deep Reinforcement Learning

Abstract

We present a control architecture for real-time adaptation and tracking of trajectories generated using a terrain-aware trajectory optimization solver. This approach enables us to circumvent the computationally exhaustive task of online trajectory optimization, and further introduces a control solution robust to systems modeled with approximated dynamics. We train a policy using deep reinforcement learning (RL) to introduce additive deviations to a reference trajectory in order to generate a feedback-based trajectory tracking system for a quadrupedal robot. We train this policy across a multitude of simulated terrains and ensure its generality by introducing training methods that avoid overfitting and convergence towards local optima. Additionally, in order to capture terrain information, we include a latent representation of the height maps in the observation space of the RL environment as a form of exteroceptive feedback. We test the performance of our trained policy by tracking the corrected set points using a model-based whole-body controller and compare it with the tracking behavior obtained without the corrective feedback in several simulation environments. We also show successful transfer of our training approach to the real physical system and further present cogent arguments in support of our framework.

Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviors

“…It should be possible to merge and grow datasets over time with recordings that target behaviors that the skill modules struggle with. Datasets could also be enriched with other sources than MoCap such as trajectory optimization over procedural terrains [52,53].…”

Deep reinforcement learning for real-world quadrupedal locomotion: a comprehensive review

“…Many studies have also considered additional information, such as trajectory generators [46,[49][50][51] , control methods [52][53][54] , motion data [10,12,55,56] , etc. Trajectory generators and control methods mainly introduce prior knowledge in the action space, narrowing the search range of DRL control policies, which greatly improves the sample efficiency under a simple reward function.…”

“…Real-Time Trajectory Adaptation for Quadrupedal Locomotion using Deep Reinforcement Learning [53] TOWR [113] , WBC [116] . Domain Randomization, Actuator Modelling, Shifting Initial Position, Changing Gravity, Actuator Torque Scaling, and Perturbing the Robot Base.…”

Direction and Trajectory Tracking Control for Nonholonomic Spherical Robot by Combining Sliding Mode Controller and Model Prediction Controller

“…[10] proposes the model-reference reinforcement learning control method for autonomous surface vehicles and obtains good results in simulation despite requiring prior training. [11] presents a reinforcement learning (RL) method to train a trajectory tracking controller for quadrupedal robots without modifying the bottom whole-body controller. For the multi-terrain control of quadrupedal robotics, [12], [13] use RL to train the robots on different terrains.…”