Towards a common implementation of reinforcement learning for multiple robotic tasks

Abstract

Mobile robots are increasingly being employed for performing complex tasks in dynamic environments. Reinforcement learning (RL) methods are recognized to be promising for specifying such tasks in a relatively simple manner. However, the strong dependency between the learning method and the task to learn is a well-known problem that restricts practical implementations of RL in robotics, often requiring major modifications of parameters and adding other techniques for each particular task. In this paper we present a practical core implementation of RL which enables the learning process for multiple robotic tasks with minimal per-task tuning or none. Based on value iteration methods, this implementation includes a novel approach for action selection, called Q-biased softmax regression (QBIASSR), which avoids poor performance of the learning process when the robot reaches new unexplored states. Our approach takes advantage of the structure of the state space by attending the physical variables involved (e.g., distances to obstacles, X , Y , θ pose, etc.), thus experienced sets of states may favor the decision-making process of unexplored or rarely-explored states. This improvement has a relevant role in reducing the tuning of the algorithm for particular tasks. Experiments with real and simulated robots, performed with the software framework also introduced here, show that our implementation is effectively able to learn different robotic tasks without tuning the learning method. Results also suggest that the combination of true online SARSA(λ) (TOSL) with QBIASSR can outperform the existing RL core algorithms in low-dimensional robotic tasks.

Mobile Robot Navigation Using Reinforcement Learning in Unknown Environments

“…At each step the bias is updated using averaged information from sets of states that share some structure with the current state s . The working of TOSL-QBIASSR is detailed in Algorithm 2 [20]:…”

“…The learning process, true online SARSA Q-biased softmax regression (TOSL-QBIASSR) [20], evolved from classical Qlearning, has attempted to tackle a wide variety of robotic tasks with minimal tuning required. A complimentary lowreward-loop evasion algorithm has been utilized to avoid local minima sequences.…”

A particle swarm optimization approach using adaptive entropy-based fitness quantification of expert knowledge for high-level, real-time cognitive robotic control

“…As described in the introduction, the two key factors of cognitive robotics, are knowledge representation and cognitive reasoning [3]. Although the problem of improving autonomy is non-trivial, it is relevant to a variety of robotic applications, for example, in humanoid robotics [4,5], human-robot interaction [6][7][8][9], Search and Rescue (SAR) [10] and multi-robot systems [11].…”

Robot Coverage Path Planning under Uncertainty Using Knowledge Inference and Hedge Algebras

“…In related studies investigations have proposed a manipulator control and theoretical ideas on using artificial neural networks with reinforcement learning for multiple robotic tasks [1,35].…”

“…Robotic systems are now ubiquitous in the manufacturing industry. Robots are capable of reliably manipulating objects using artificial intelligence techniques, which allows a machine to determine how a task can be completed successfully [1]. However, when employed in the manufacturing process, robots are pre-programmed with limited or no decision-making capability.…”