Dynamic heuristic acceleration of linearly approximated SARSA($$\lambda $$): using ant colony optimization to learn heuristics dynamically

Abstract

Heuristically accelerated reinforcement learning (HARL) is a new family of algorithms that combines the advantages of reinforcement learning (RL) with the advantages of heuristic algorithms. To achieve this, the action selection strategy of the standard RL algorithm is modified to take into account a heuristic running in parallel with the RL process. This paper presents two approximated HARL algorithms that make use of pheromone trails to improve the behaviour of linearly approximated SARSA(λ) by dynamically learning a heuristic function through the pheromone trails. The proposed dynamic algorithms are evaluated in comparison to linearly approximated SARSA(λ), and heuristically accelerated SARSA(λ) using a static heuristic in three benchmark scenarios: the mountain car, the mountain car 3D and the maze scenarios.

Discrete-Event Simulation-Based Q-Learning Algorithm Applied to Financial Leverage Effect

“…When it receives a new tuple (s, r, d), it will return an action following a policy depending on the implemented algorithm ( -greedy for example). The model has a Q state variable which is an SxA dimension matrix for implementing the learning algorithm (Q-learning or SARSA [4] for example). When convergence is reached (depending on the values of Q), the model becomes passive and no longer responds to the environment.…”

Distributed Multi-Ant Colony System Algorithm using Raspberry Pi Cluster for Travelling Salesman Problem

“…The mechanism of the ACS algorithm is inspired by the natural behavior of biological ant colonies [2] [3]. The ACS algorithm is built on notation of the reinforcement learning concept [4]. It uses ants as agents to manipulate environment via use of pheromone trails to find the shortest path.…”