Model-Free Reinforcement Learning with Ensemble for a Soft Continuum Robot Arm

Morimoto, Ryota; Nishikawa, Satoshi; Niiyama, Ryuma; Kuniyoshi, Yasuo

doi:10.1109/robosoft51838.2021.9479340

Cited by 31 publications

(18 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Model- [1,6] and learningbased [33,12,29] controllers have also proven successful, as well as hybrid policy designs [61,2,26]. Zhu et al [64] consider an origami-like robot with various design configurations that all inform policy optimization, and Morimoto et al [38] employ the soft actor-critic algorithm [23] for reaching tasks. Related, Vikas et al [59] present a modular approach to designing 3Dprinted motor-tendon soft robots that can be readily fabricated, and a model-free algorithm for learning the corresponding control policy.…”

Section: Related Workmentioning

confidence: 99%

Soft Robots Learn to Crawl: Jointly Optimizing Design and Control with Sim-to-Real Transfer

Schaff¹,

Sedal²,

Walter³

2022

Preprint

View full text Add to dashboard Cite

This work provides a complete framework for the simulation, co-optimization, and sim-to-real transfer of the design and control of soft legged robots. The compliance of soft robots provides a form of "mechanical intelligence"-the ability to passively exhibit behaviors that would otherwise be difficult to program. Exploiting this capacity requires careful consideration of the coupling between mechanical design and control. Cooptimization provides a promising means to generate sophisticated soft robots by reasoning over this coupling. However, the complex nature of soft robot dynamics makes it difficult to provide a simulation environment that is both sufficiently accurate to allow for sim-to-real transfer, while also being fast enough for contemporary co-optimization algorithms. In this work, we show that finite element simulation combined with recent model order reduction techniques provide both the efficiency and the accuracy required to successfully learn effective soft robot design-control pairs that transfer to reality. We propose a reinforcement learning-based framework for co-optimization and demonstrate successful optimization, construction, and zeroshot sim-to-real transfer of several soft crawling robots. Our learned robot outperforms an expert-designed crawling robot, showing that our approach can generate novel, high-performing designs even in well-understood domains.

show abstract

Section: Related Workmentioning

confidence: 99%

Soft Robots Learn to Crawl: Jointly Optimizing Design and Control with Sim-to-Real Transfer

Schaff¹,

Sedal²,

Walter³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Model- [1,6] and learningbased [33,12,29] controllers have also proven successful, as well as hybrid policy designs [62,2,26]. Zhu et al [65] consider an origami-like robot with various design configurations that all inform policy optimization, and Morimoto et al [38] employ the soft actor-critic algorithm [23] for reaching tasks. Related, Vikas et al [60] present a modular approach to designing 3Dprinted motor-tendon soft robots that can be readily fabricated, and a model-free algorithm for learning the corresponding Fig.…”

Section: Related Workmentioning

confidence: 99%

Soft Robots Learn to Crawl: Jointly Optimizing Design and Control with Sim-to-Real Transfer

Schaff¹,

Sedal²,

Walter³

2022

Robotics: Science and Systems XVIII

View full text Add to dashboard Cite

This work provides a complete framework for the simulation, co-optimization, and sim-to-real transfer of the design and control of soft legged robots. The compliance of soft robots provides a form of "mechanical intelligence"-the ability to passively exhibit behaviors that would otherwise be difficult to program. Exploiting this capacity requires careful consideration of the coupling between mechanical design and control. Cooptimization provides a promising means to generate sophisticated soft robots by reasoning over this coupling. However, the complex nature of soft robot dynamics makes it difficult to achieve a simulation environment that is both sufficiently accurate to allow for sim-to-real transfer and fast enough for contemporary co-optimization algorithms. In this work, we describe a modularized model order reduction algorithm that significantly improves the efficiency of finite element simulation, while preserving the accuracy required to successfully learn effective soft robot design-control pairs that transfer to reality. We propose a reinforcement learning-based framework for co-optimization and demonstrate successful optimization, construction, and zero-shot sim-to-real transfer of several soft crawling robots. Our learned robot outperforms an expert-designed crawling robot, showing that our approach can generate novel, high-performing designs even in well-understood domains.

show abstract

“…Therefore, data-driven control methods are considered useful for soft robots because of modeling difficulty, and the application of reinforcement learning has been proposed ( Bhagat et al, 2019 ). Data-driven methods are often based on machine learning and include sampling data by actually moving the robot and modeling it using machine learning ( Bruder et al, 2019 ; Buchler et al, 2018 ; George Thuruthel et al, 2017 ; Giorelli et al, 2015 ; Lee et al, 2017 ; Rolf and Steil, 2014 ; Thuruthel et al, 2017 ); and learning directly implemented on the controller by moving the robot and using reinforcement learning ( Chattopadhyay et al, 2018 ; Morimoto et al, 2021 ).…”

Section: Related Workmentioning

confidence: 99%

“…Model-based reinforcement learning is feasible to some extent for continuum robot arms, which can be modeled and are relatively simple in structure and materials used. However, if the robot moves in 3D space or has a large number of actuators, regardless of whether the model is created by humans or acquired by learning using data-driven methods, the differences between the real and simulation robots increase, and the learning is adversely affected ( Morimoto et al, 2021 ). Therefore, for selecting a reinforcement learning method that can be applied to many continuum robot arms, a model-free reinforcement learning method is preferable in which the robot model is neither provided by a user nor the forward model acquired through learning.…”

Section: Related Workmentioning

confidence: 99%

“…Accordingly, learning using reinforcement learning can potentially solve the problem of soft robot control by developing data-driven control methods ( Bhagat et al, 2019 ). Several studies have been conducted on reaching tasks using reinforcement learning ( You et al, 2017 ; Zhang et al, 2017 ; Chattopadhyay et al, 2018 ; Satheeshbabu et al, 2019 ; Fetchrobotics, 2020 ; Morimoto et al, 2021 ). However, the research on soft robots and continuum robot arms, especially with regard to reinforcement learning, depends on the continuum robot arms owned by each research group, and there is no investigation into comparison with conventional “rigid” robots.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Characterization of continuum robot arms under reinforcement learning and derived improvements

et al. 2022

Self Cite

View full text Add to dashboard Cite

In robotics, soft continuum robot arms are a promising prospect owing to their redundancy and passivity; however, no comprehensive study exists that examines their characteristics compared to rigid manipulators. In this study, we examined the advantages of a continuum robot arm as compared to a typical and rigid seven-degree-of-freedom (7-DoF) robot manipulator in terms of performing various tasks through reinforcement learning. We conducted simulations for tasks with different characteristics that require control over position and force. Common tasks in robot manipulators, such as reaching, crank rotation, object throwing, and peg-in-hole were considered. The initial conditions of the robot and environment were randomized, aiming for evaluations including robustness. The results indicate that the continuum robot arm excels in the crank-rotation task, which is characterized by uncertainty in environmental conditions and cumulative rewards. However, the rigid robot arm learned better motions for the peg-in-hole task than the other tasks, which requires fine motion control of the end-effector. In the throwing task, the continuum robot arm scored well owing to its good handling of anisotropy. Moreover, we developed a reinforcement-learning method based on the comprehensive experimental results. The proposed method successfully improved the motion learning of a continuum robot arm by adding a technique to regulate the initial state of the robot. To the best of our knowledge, ours is the first reinforcement-learning experiment with multiple tasks on a single continuum robot arm and is the first report of a comparison between a single continuum robot arm and rigid manipulator on a wide range of tasks. This simulation study can make a significant contribution to the design of continuum arms and specification of their applications, and development of control and reinforcement learning methods.

show abstract

Model-Free Reinforcement Learning with Ensemble for a Soft Continuum Robot Arm

Cited by 31 publications

References 23 publications

Soft Robots Learn to Crawl: Jointly Optimizing Design and Control with Sim-to-Real Transfer

Soft Robots Learn to Crawl: Jointly Optimizing Design and Control with Sim-to-Real Transfer

Soft Robots Learn to Crawl: Jointly Optimizing Design and Control with Sim-to-Real Transfer

Characterization of continuum robot arms under reinforcement learning and derived improvements

Contact Info

Product

Resources

About