Task-Oriented Deep Reinforcement Learning for Robotic Skill Acquisition and Control

Xiang, Guofei; Su, Jianbo

doi:10.1109/tcyb.2019.2949596

Cited by 37 publications

(12 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Related Work in Entropy Regularization: Some of the previous works in RL literature [14]- [20], [30], [31] either use entropy as a regularization term (− log µ(a t |s t )) [14], [15] to the instantaneous cost function c(s t , a t , s t+1 ) or maximize the entropy (− a µ(a|s) log µ(a|s)) [16]- [18] associated only to the stochastic policy under constraints on the cost J. This results into benefits such as better exploration, overcoming the effect of noise w t in the instantaneous cost c t and obtaining faster convergence.…”

Section: Preliminariesmentioning

confidence: 99%

Parameterized MDPs and Reinforcement Learning Problems—A Maximum Entropy Principle-Based Framework

Srivastava

Salapaka

2022

IEEE Trans. Cybern.

View full text Add to dashboard Cite

We present a framework to address a class of sequential decision making problems. Our framework features learning the optimal control policy with robustness to noisy data, determining the unknown state and action parameters, and performing sensitivity analysis with respect to problem parameters. We consider two broad categories of sequential decision making problems modeled as infinite horizon Markov Decision Processes (MDPs) with (and without) an absorbing state. The central idea underlying our framework is to quantify exploration in terms of the Shannon Entropy of the trajectories under the MDP and determine the stochastic policy that maximizes it while guaranteeing a low value of the expected cost along a trajectory. This resulting policy enhances the quality of exploration early on in the learning process, and consequently allows faster convergence rates and robust solutions even in the presence of noisy data as demonstrated in our comparisons to popular algorithms such as Q-learning, Double Q-learning and entropy regularized Soft Q-learning. The framework extends to the class of parameterized MDP and RL problems, where states and actions are parameter dependent, and the objective is to determine the optimal parameters along with the corresponding optimal policy. Here, the associated cost function can possibly be non-convex with multiple poor local minima. Simulation results applied to a 5G small cell network problem demonstrate successful determination of communication routes and the small cell locations. We also obtain sensitivity measures to problem parameters and robustness to noisy environment data.

show abstract

Section: Preliminariesmentioning

confidence: 99%

Parameterized MDPs and Reinforcement Learning Problems—A Maximum Entropy Principle-Based Framework

Srivastava

Salapaka

2022

IEEE Trans. Cybern.

View full text Add to dashboard Cite

show abstract

“…According to different types of tasks, task-oriented programming by demonstration has various representation methods. In recent years, task-oriented PBD methods based on reinforcement learning (RL) [20] have received more attention. This method offers a new paradigm for acquiring skill by maximizing the overall reward.…”

Section: B Task-oriented Programming By Demonstrationmentioning

confidence: 99%

A Skill Programming Method Based on Assembly Motion Primitive for Modular Assembly System

et al. 2021

View full text Add to dashboard Cite

To improve the programming efficiency of automatic assembly system, a novel skill programming framework based on task learning is proposed for modular assembly system in this paper. In this framework, the motion sequence of assembly skills can be modeled by demonstration data. And the assembly task is represented hierarchically. A complete assembly process of a part is divided into several skills, and each skill is divided into several sequential assembly motion primitives (AMP) of multiple modules. Then, a learning method of assembly motion sequence based on Hidden Markov Model is proposed, and the maximum probability method is used to generate the optimal sequential AMP. Each AMP is input to the assembly system in the form of instruction to complete the assembly. Aiming at the problem of accurate positioning and trajectory planning, visual guidance and direct teaching method are used to settle this problem. To evaluate the viability of the proposed framework, a customized modular assembly system is used to acquire the demonstration data, and a graphical user interface (GUI) software is designed. Five assembly skills are learned. Experimental are conducted to validate the effectiveness of the proposed method.

show abstract

“…The control method has the characteristics of good generality, strong adaptability and easy extension. Xiang and Su [10] proposes an effective model-free off-policy actor-critic algorithm by integrating task reward and task-oriented guiding reward, and applies it to the skill acquisition and continuous control of the robot. The agent can explore the environment more consciously so as to realize the sampling efficiency.…”

Section: Introductionmentioning

confidence: 99%

Hierarchical Reinforcement Learning Considering Stochastic Wind Disturbance for Power Line Maintenance Robot

Zheng

2021

Preprint

View full text Add to dashboard Cite

Robot intelligence includes motion intelligence and cognitive intelligence. Aiming at the motion intelligence, a hierarchical reinforcement learning architecture considering stochastic wind disturbance is proposed for the decision-making of the power line maintenance robot with autonomous operation. This architecture uses the prior information of the mechanism knowledge and empirical data to improve the safety and efficiency of the robot operation. In this architecture, the high-level policy selection and the low-level motion control at global and local levels are considered comprehensively under the condition of stochastic wind disturbance. Firstly, the operation task is decomposed into three sub-policies: global obstacle avoidance, local approach and local tightening, and each sub-policy is learned. Then, a master policy is learned to select the operation sub-policy in the current state. The dual deep Q network algorithm is used for the master policy, while the deep deterministic policy gradient algorithm is used for the operation policy. In order to improve the training efficiency, the global obstacle avoidance sub-policy takes the random forest composed of dynamic environmental decision tree as the expert algorithm for imitation learning. The architecture is applied to a power line maintenance scenario, the state function and reward function of each policy are designed, and all policies are trained in an asynchronous and parallel computing environment. It is proved that this architecture can realize stable and safe autonomous operating decision for the power line maintenance robot subjected to stochastic wind disturbance.

show abstract

Task-Oriented Deep Reinforcement Learning for Robotic Skill Acquisition and Control

Cited by 37 publications

References 23 publications

Parameterized MDPs and Reinforcement Learning Problems—A Maximum Entropy Principle-Based Framework

Parameterized MDPs and Reinforcement Learning Problems—A Maximum Entropy Principle-Based Framework

A Skill Programming Method Based on Assembly Motion Primitive for Modular Assembly System

Hierarchical Reinforcement Learning Considering Stochastic Wind Disturbance for Power Line Maintenance Robot

Contact Info

Product

Resources

About