Stochastic Abstract Policies: Generalizing Knowledge to Improve Reinforcement Learning

Abstract

Reinforcement learning (RL) enables an agent to learn behavior by acquiring experience through trial-and-error interactions with a dynamic environment. However, knowledge is usually built from scratch and learning to behave may take a long time. Here, we improve the learning performance by leveraging prior knowledge; that is, the learner shows proper behavior from the beginning of a target task, using the knowledge from a set of known, previously solved, source tasks. In this paper, we argue that building stochastic abstract policies that generalize over past experiences is an effective way to provide such improvement and this generalization outperforms the current practice of using a library of policies. We achieve that contributing with a new algorithm, AbsProb-PI-multiple and a framework for transferring knowledge represented as a stochastic abstract policy in new RL tasks. Stochastic abstract policies offer an effective way to encode knowledge because the abstraction they provide not only generalizes solutions but also facilitates extracting the similarities among tasks. We perform experiments in a robotic navigation environment and analyze the agent's behavior throughout the learning process and also assess the transfer ratio for different amounts of source tasks. We compare our method with the transfer of a library of policies, and experiments show that the use of a generalized policy produces better results by more effectively guiding the agent when learning a target task.

Bootstrapping $Q$ -Learning for Robotics From Neuro-Evolution Results

“…Furthermore, options learned in a particular setup can be transferred to a new task or domain if they are defined in a problem independent space [29]. Koga et al have proposed an approach to learn abstract policies in order to increase the generalization ability [30,31]. Riano and McGinnity proposed to represent sequences of actions with a finite state automaton and to explore its structure with evolutionary algorithms [32].…”

Reusing Source Task Knowledge via Transfer Approximator in Reinforcement Transfer Learning

“…The online transfer framework proposed in [11] uses advice as transferred knowledge, but the difference between tasks was not emphasized. The work [12] treated the source task and the target task as one single problem, so no consideration on the inter-task mapping was given. To avoid the need of inter-task mapping, Laflamm [13] chose the source task and the target task from the same domain with the same state and action space, and then compared three existing transfer learning methods on the Mario AI domain.…”

Knowledge reuse for deep reinforcement learning.

“…While they select a whole policy that they follow for a certain amount of time during training, we estimate the usefulness of policies and mix the policies during training instead of following a single policy per episode. Koga, Freire and Costa (2015) propose to blend multiple policies into a single abstract policy, which is used at the beginning of learning in any new task (whether the new task is similar to the source tasks or not). In spite of following a similar idea, DE-CAF stores multiple concrete policies, and selects only the most promising ones by taking similarity with the target task into consideration.…”

“…• Another important aspect for making gained insights more valuable is generating better abstractions of this knowledge to make it more applicable to a wider range of problems without having to be sure about every detail, as for example shown in Koga, Freire and Costa (2015). Another approach could be to rethink the way we are looking at state values at the moment by factoring in more information than simply the state representation as has been investigated with Universal Value Function Approximators introduced by Schaul et al (2015).…”