Transferring knowledge as heuristics in reinforcement learning: A case-based approach

Bianchi, Reinaldo A. C.; Celiberto, Luiz A.; Santos, Paulo E.; Matsuura, Jackson P.; Mántaras, Ramon López de

doi:10.1016/j.artint.2015.05.008

Cited by 59 publications

(27 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Future work includes the addition of advanced techniques for boosting the learning process in robotics, e.g., by constructing a model (model-based RL), such as prioritized sweeping, by human interaction as teaching, and by transferring (Taylor & Stone 2009), (Barrett, Taylor & Stone 2010) and (Bianchi, Celiberto, Santos, Matsuura & de Mantaras 2015) from simulations to real robots. We also believe that these techniques will contribute to decrease the variance of the learning processes seen here, which sometimes have produced inconclusive results, especially in episodic tasks.…”

Section: Discussionmentioning

confidence: 99%

Towards a common implementation of reinforcement learning for multiple robotic tasks

Martínez-Tenor

Fernández-Madrigal

Cruz‐Martín

et al. 2018

Expert Systems with Applications

View full text Add to dashboard Cite

Mobile robots are increasingly being employed for performing complex tasks in dynamic environments. Reinforcement learning (RL) methods are recognized to be promising for specifying such tasks in a relatively simple manner. However, the strong dependency between the learning method and the task to learn is a well-known problem that restricts practical implementations of RL in robotics, often requiring major modifications of parameters and adding other techniques for each particular task. In this paper we present a practical core implementation of RL which enables the learning process for multiple robotic tasks with minimal per-task tuning or none. Based on value iteration methods, this implementation includes a novel approach for action selection, called Q-biased softmax regression (QBIASSR), which avoids poor performance of the learning process when the robot reaches new unexplored states. Our approach takes advantage of the structure of the state space by attending the physical variables involved (e.g., distances to obstacles, X , Y , θ pose, etc.), thus experienced sets of states may favor the decision-making process of unexplored or rarely-explored states. This improvement has a relevant role in reducing the tuning of the algorithm for particular tasks. Experiments with real and simulated robots, performed with the software framework also introduced here, show that our implementation is effectively able to learn different robotic tasks without tuning the learning method. Results also suggest that the combination of true online SARSA(λ) (TOSL) with QBIASSR can outperform the existing RL core algorithms in low-dimensional robotic tasks.

show abstract

Section: Discussionmentioning

confidence: 99%

Towards a common implementation of reinforcement learning for multiple robotic tasks

Martínez-Tenor

Fernández-Madrigal

Cruz‐Martín

et al. 2018

Expert Systems with Applications

View full text Add to dashboard Cite

show abstract

“…In the TBO, 50% of bees with nectar amounts that rank in the half top of all bees are designated as worker, while the others are scout. On the basis of the ε-Greedy rule [31], the scouts' actions are based on the proportion of Q-value in the current status. As for the controlled variable x i , one behavior of each scout is chosen as…”

Section: Action Policymentioning

confidence: 99%

“…Also, reinforcement learning (RL) can be accelerated by knowledge conversion [29], and agents learn new tasks faster and interact less with the environment. As a consequence, knowledge transfer reinforcement learning (KTRL) has been developed [30] through combining AI and behavior psychology [31] and is divided into behavior shift and information shift.…”

Section: Introductionmentioning

confidence: 99%

Reactive Power Optimization of Large-Scale Power Systems: A Transfer Bees Optimizer Application

Zhang

Yang

et al. 2019

Processes

View full text Add to dashboard Cite

A novel transfer bees optimizer for reactive power optimization in a high-power system was developed in this paper. Q-learning was adopted to construct the learning mode of bees, improving the intelligence of bees through task division and cooperation. Behavior transfer was introduced, and prior knowledge of the source task was used to process the new task according to its similarity to the source task, so as to accelerate the convergence of the transfer bees optimizer. Moreover, the solution space was decomposed into multiple low-dimensional solution spaces via associated state-action chains. The transfer bees optimizer performance of reactive power optimization was assessed, while simulation results showed that the convergence of the proposed algorithm was more stable and faster, and the algorithm was about 4 to 68 times faster than the traditional artificial intelligence algorithms.

show abstract

“…In this sense, the learned pheromone distribution could help into defining a function to generate new trajectories and therefore improve the results of BO. The learning of heuristic functions is also important towards transfer learning [see the contribution of Weiss et al (2016) for a survey of transfer learning applied on several domains] in the reinforcement learning domain, as also shown by Bianchi et al (2015). With little modification, the heuristic function learned with AP-HARL could be simply used as a model for a particular task or problem and then re-used in a related problem.…”

Section: Related Workmentioning

confidence: 99%

Dynamic heuristic acceleration of linearly approximated SARSA($$\lambda $$): using ant colony optimization to learn heuristics dynamically

Bromuri

2019

J Heuristics

View full text Add to dashboard Cite

Heuristically accelerated reinforcement learning (HARL) is a new family of algorithms that combines the advantages of reinforcement learning (RL) with the advantages of heuristic algorithms. To achieve this, the action selection strategy of the standard RL algorithm is modified to take into account a heuristic running in parallel with the RL process. This paper presents two approximated HARL algorithms that make use of pheromone trails to improve the behaviour of linearly approximated SARSA(λ) by dynamically learning a heuristic function through the pheromone trails. The proposed dynamic algorithms are evaluated in comparison to linearly approximated SARSA(λ), and heuristically accelerated SARSA(λ) using a static heuristic in three benchmark scenarios: the mountain car, the mountain car 3D and the maze scenarios.

show abstract

Transferring knowledge as heuristics in reinforcement learning: A case-based approach

Cited by 59 publications

References 32 publications

Towards a common implementation of reinforcement learning for multiple robotic tasks

Towards a common implementation of reinforcement learning for multiple robotic tasks

Reactive Power Optimization of Large-Scale Power Systems: A Transfer Bees Optimizer Application

Dynamic heuristic acceleration of linearly approximated SARSA($$\lambda $$): using ant colony optimization to learn heuristics dynamically

Contact Info

Product

Resources

About