Towards Safe Control of Continuum Manipulator Using Shielded Multiagent Reinforcement Learning

Ji, Guanglin; Yan, Junyan; Du, Jingxin; Yan, Wanquan; Chen, Jibiao; Lu, Yongkang; Rojas, Juan; Cheng, Shing Shin

doi:10.1109/lra.2021.3097660

Cited by 24 publications

(11 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1. Multi-agent Deep Q-Network (MADQN) [40]: This is the multi-agent version of Deep Q-Network (DQN) [25], which is an off-policy RL method by applying a deep neural network to approximate the value function and an experience replay buffer to break the correlations between samples to stabilize the training.…”

Section: Comparison With the State-of-the-art Benchmarksmentioning

confidence: 99%

Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic

Zhou

Chen

Yan

et al. 2022

Auton. Intell. Syst.

View full text Add to dashboard Cite

Autonomous driving has attracted significant research interests in the past two decades as it offers many potential benefits, including releasing drivers from exhausting driving and mitigating traffic congestion, among others. Despite promising progress, lane-changing remains a great challenge for autonomous vehicles (AV), especially in mixed and dynamic traffic scenarios. Recently, reinforcement learning (RL) has been widely explored for lane-changing decision makings in AVs with encouraging results demonstrated. However, the majority of those studies are focused on a single-vehicle setting, and lane-changing in the context of multiple AVs coexisting with human-driven vehicles (HDVs) have received scarce attention. In this paper, we formulate the lane-changing decision-making of multiple AVs in a mixed-traffic highway environment as a multi-agent reinforcement learning (MARL) problem, where each AV makes lane-changing decisions based on the motions of both neighboring AVs and HDVs. Specifically, a multi-agent advantage actor-critic (MA2C) method is proposed with a novel local reward design and a parameter sharing scheme. In particular, a multi-objective reward function is designed to incorporate fuel efficiency, driving comfort, and the safety of autonomous driving. A comprehensive experimental study is made that our proposed MARL framework consistently outperforms several state-of-the-art benchmarks in terms of efficiency, safety, and driver comfort.

show abstract

Section: Comparison With the State-of-the-art Benchmarksmentioning

confidence: 99%

Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic

Zhou

Chen

Yan

et al. 2022

Auton. Intell. Syst.

View full text Add to dashboard Cite

show abstract

“…They range from using continuum robot arms with one segment ( Chattopadhyay et al, 2018 ; Satheeshbabu et al, 2019 , 2020 ) to two ( Yang et al, 2019 ), three ( Zhang et al, 2017 ), and four ( You et al, 2017 ) segments. There are also other studies that use multi-agent reinforcement learning in which each actuator of a multi-degree-of-freedom arm is considered as one agent ( Ansari et al, 2018 ; Perrusquía et al, 2020 ; Ji et al, 2021 ). Furthermore, there are studies that use reinforcement learning for the reaching component of the hierarchical control of tasks involving interactions with the environment of a continuum robot arm ( Jiang et al, 2021 ).…”

Section: Related Workmentioning

confidence: 99%

Characterization of continuum robot arms under reinforcement learning and derived improvements

et al. 2022

View full text Add to dashboard Cite

In robotics, soft continuum robot arms are a promising prospect owing to their redundancy and passivity; however, no comprehensive study exists that examines their characteristics compared to rigid manipulators. In this study, we examined the advantages of a continuum robot arm as compared to a typical and rigid seven-degree-of-freedom (7-DoF) robot manipulator in terms of performing various tasks through reinforcement learning. We conducted simulations for tasks with different characteristics that require control over position and force. Common tasks in robot manipulators, such as reaching, crank rotation, object throwing, and peg-in-hole were considered. The initial conditions of the robot and environment were randomized, aiming for evaluations including robustness. The results indicate that the continuum robot arm excels in the crank-rotation task, which is characterized by uncertainty in environmental conditions and cumulative rewards. However, the rigid robot arm learned better motions for the peg-in-hole task than the other tasks, which requires fine motion control of the end-effector. In the throwing task, the continuum robot arm scored well owing to its good handling of anisotropy. Moreover, we developed a reinforcement-learning method based on the comprehensive experimental results. The proposed method successfully improved the motion learning of a continuum robot arm by adding a technique to regulate the initial state of the robot. To the best of our knowledge, ours is the first reinforcement-learning experiment with multiple tasks on a single continuum robot arm and is the first report of a comparison between a single continuum robot arm and rigid manipulator on a wide range of tasks. This simulation study can make a significant contribution to the design of continuum arms and specification of their applications, and development of control and reinforcement learning methods.

show abstract

“…For continuum robots (15) with disturbance d(t), which is bounded and whose derivative exists, the dynamic model can be expressed as B q (q)q + C q (q, q) q + K q q + D q (q) q = τ − d(t). (21) Then a nominal SMC controller can be designed as…”

Section: A Tailored Sliding Mode Control Schemementioning

confidence: 99%

“…Moreover, when considering the uncertainties of the continuum robots, including both the intervention from the environment and the parameter perturbation of the system, it is hard to achieve high precision control for continuum robots in a complex uncertain environment. To this end, G. Ji et al presented the superiority of controlling a cable-driven continuum robot using a multi-agent DQN with shielding [21]. Although their work considered the external payload, soft and hard interference, they did not consider the internal uncertainties.…”

Section: Introductionmentioning

confidence: 99%

A Novel Disturbance-Rejection Control Framework for Cable-Driven Continuum Robots With Improved State Parameterizations

et al. 2022

View full text Add to dashboard Cite

A soft, or continuum, robot has great potential in many practical fields with confined space , such as manufacturing system, electrical industry and other, due to its flexibility and benign bending attribute. However, its innate nonlinear dynamics and infinite degrees of freedom pose challenges both in modeling and control. How to build a proper model, which can be used to analyze the characteristics of soft robots, and how to design a robustly stabilized controller, are still open issues.In this work, we build a model for cable-driven continuum robots using a piecewise constant-curvature method. An improved state parameterization is proposed to avoid singularities, which produce pathological behaviors mainly on straight configurations of continuum robots. Then, a novel disturbance-rejection control framework is proposed with a tailored sliding mode controller to achieve stabilized control. Aiming at better robustness, we lump all system uncertainties, including unmodeled dynamics, unknown external disturbances, and unknown parameter variations, together as an extended state of the system. A linear extended state observer is then proposed to estimate the total uncertainties online, enabling compensation for their negative effects. Stability analysis and numerical simulations are carried out to show the feasibility and effectiveness of the proposed approaches.

show abstract

Towards Safe Control of Continuum Manipulator Using Shielded Multiagent Reinforcement Learning

Cited by 24 publications

References 22 publications

Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic

Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic

Characterization of continuum robot arms under reinforcement learning and derived improvements

A Novel Disturbance-Rejection Control Framework for Cable-Driven Continuum Robots With Improved State Parameterizations

Contact Info

Product

Resources

About