The Ingredients of Real-World Robotic Reinforcement Learning

Zhu, Henry; Yu, Justin; Gupta, Abhishek; Shah, Dhruv; Hartikainen, Kristian; Singh, Avi; Kumar, Vikash; Levine, Sergey

doi:10.48550/arxiv.2004.12570

Cited by 19 publications

(30 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reinforcement Learning is one of the most favored methods for recent robot learning fields; it results in robots that are autonomous and flexible in performing several specific tasks. Although the application in practice of reinforcement learning algorithms to realworld robots is still difficult and challenging [3], there are still some applications that are implemented in practice, especially in the field of robotics [4][5][6][7][8][9]. In the recent decade, reinforcement learning has been applied to leg-robots in a variety of ways [10][11][12][13][14].…”

Section: Introductionmentioning

confidence: 99%

GCTD3: Modeling of Bipedal Locomotion by Combination of TD3 Algorithms and Graph Convolutional Network

2022

View full text Add to dashboard Cite

In recent years, there has been a lot of research using reinforcement learning algorithms to train 2-legged robots to move, but there are still many challenges. The authors propose the GCTD3 method, which takes the idea of using Graph Convolutional Networks to represent the kinematic link features of the robot, and combines this with the Twin-Delayed Deep Deterministic Policy Gradient algorithm to train the robot to move. Graph Convolutional Networks are very effective in graph-structured problems such as the connection of the joints of the human-like robots. The GCTD3 method shows better results on the motion trajectories of the bipedal robot joints compared with other reinforcement learning algorithms such as Twin-Delayed Deep Deterministic Policy Gradient, Deep Deterministic Policy Gradient and Soft Actor Critic. This research is implemented on a 2-legged robot model with six independent joint coordinates through the Robot Operating System and Gazebo simulator.

show abstract

Section: Introductionmentioning

confidence: 99%

GCTD3: Modeling of Bipedal Locomotion by Combination of TD3 Algorithms and Graph Convolutional Network

2022

View full text Add to dashboard Cite

show abstract

“…Reinforcement learning (RL) offers an appealing opportunity to enable autonomous acquisition of complex behaviors for interactive agents. Despite recent RL successes on robots [26,33,25,27,34,22,31,23,14], several challenges exist that inhibit wider adoption of reinforcement learning for robotics [47]. One of the major challenges to the autonomy of current reinforcement learning algorithms, particularly in robotics, is the assumption that each trial starts from an initial state drawn from a specific state distribution in the environment.…”

Section: Introductionmentioning

confidence: 99%

“…Eliminating or minimizing the algorithmic reliance on the reset mechanisms can enable more autonomous learning, and in turn it will allow agents to scale to broader and harder set of tasks. To address these challenges, some recent works have developed reinforcement learning algorithms that can effectively learn with minimal resets to the initial distribution [19,6,47,42,14]. We provide a formal problem definition that encapsulates and sheds light on the general setting addressed by these prior methods, which we refer to as the persistent reinforcement learning problem.…”

Section: Introductionmentioning

confidence: 99%

“…In this setting, the agent must persistently learn and interact with the environment with minimal human interventions, as shown in Figure 1. Conventional episodic RL algorithms often fail to solve the task entirely in this setting, as shown by Zhu et al [47] and Figure 2. This is because these methods rely on the ability to sample the initial state distribution arbitrarily.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Autonomous Reinforcement Learning via Subgoal Curricula

Sharma¹,

Gupta²,

Levine³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Reinforcement learning (RL) promises to enable autonomous acquisition of complex behaviors for diverse agents. However, the success of current reinforcement learning algorithms is predicated on an often under-emphasised requirement -each trial needs to start from a fixed initial state distribution. Unfortunately, resetting the environment to its initial state after each trial requires substantial amount of human supervision and extensive instrumentation of the environment which defeats the purpose of autonomous reinforcement learning. In this work, we propose Value-accelerated Persistent Reinforcement Learning (VaPRL), which generates a curriculum of initial states such that the agent can bootstrap on the success of easier tasks to efficiently learn harder tasks. The agent also learns to reach the initial states proposed by the curriculum, minimizing the reliance on human interventions into the learning. We observe that VaPRL reduces the interventions required by three orders of magnitude compared to episodic RL while outperforming prior state-of-the art methods for reset-free RL both in terms of sample efficiency and asymptotic performance on a variety of simulated robotics problems 1 .

show abstract

“…Reinforcement learning (RL) has achieved impressive success in a variety of domains: the games of Go [1] and Atari [2,3]. These successes have led to an interest in solving real-world problems such as robotics [4,5], automation driving [6,7], and unmanned aerial vehicle [8,9]. While providing high performance, RL algorithms suffer from a lack of safety certificates required for safety-critical systems.…”

Section: Introductionmentioning

confidence: 99%

Stability-Certified Reinforcement Learning via Spectral Normalization

Takase¹,

Yoshikawa²,

Mariyama³

et al. 2020

Preprint

View full text Add to dashboard Cite

In this article, two types of methods from different perspectives based on spectral normalization are described for ensuring the stability of the system controlled by a neural network. The first one is that the L2 gain of the feedback system is bounded less than 1 to satisfy the stability condition derived from the small-gain theorem. While explicitly including the stability condition, the first method may provide an insufficient performance on the neural network controller due to its strict stability condition. To overcome this difficulty, the second one is proposed, which improves the performance while ensuring the local stability with a larger region of attraction. In the second method, the stability is ensured by solving linear matrix inequalities after training the neural network controller. The spectral normalization proposed in this article improves the feasibility of the a-posteriori stability test by constructing tighter local sectors. The numerical experiments show that the second method provides enough performance compared with the first one while ensuring enough stability compared with the existing reinforcement learning algorithms.

show abstract

The Ingredients of Real-World Robotic Reinforcement Learning

Cited by 19 publications

References 18 publications

GCTD3: Modeling of Bipedal Locomotion by Combination of TD3 Algorithms and Graph Convolutional Network

GCTD3: Modeling of Bipedal Locomotion by Combination of TD3 Algorithms and Graph Convolutional Network

Autonomous Reinforcement Learning via Subgoal Curricula

Stability-Certified Reinforcement Learning via Spectral Normalization

Contact Info

Product

Resources

About