Transfer Learning via Multiple Inter-task Mappings

Abstract

Abstract. In this paper we investigate using multiple mappings for transfer learning in reinforcement learning tasks. We propose two different transfer learning algorithms that are able to manipulate multiple inter-task mappings for both model-learning and model-free reinforcement learning algorithms. Both algorithms incorporate mechanisms to select the appropriate mappings, helping to avoid the phenomenon of negative transfer. The proposed algorithms are evaluated in the Mountain Car and Keepaway domains. Experimental results show that the use of multiple inter-task mappings can significantly boost the performance of transfer learning methodologies, relative to using a single mapping or learning without transfer.

Evolutionary Policy Transfer and Search Methods for Boosting Behavior Quality: RoboCup Keep-Away Case Study

“…Furthermore, keeper-team behaviors evolved and transferred to increasingly complex keep-away tasks were found to be comparable in average task performance to keeper-team policies derived with RL methods and policy transfer (Stone et al, 2006b;Whiteson and Stone, 2006). Verbancsics and Stanley (2010) also used HyperNEAT to demonstrate successful transfer of collective behaviors between Knight's Joust, which is a multiagent predator-prey task variant (Taylor et al, 2010), and keep-away soccer tasks. The efficacy of this policy transfer method was supported by improved task performance on target tasks given further behavior evolution.…”

“…These variants are a fitness function (objective-based search), behavioral diversity maintenance (novelty search), genotypic diversity maintenance (Section 3), and both genotypic and behavioral diversity maintenance hybridized with objective-based search. RoboCup keep-away was selected as it is a well-established multiagent (robot) experimental platform (Taylor et al, 2010). This study thus evaluates various evolutionary search methods coupled with policy transfer as a means to increase the quality of evolved collective (keep-away) behaviors.…”

“…Subsequent work by Taylor et al (2010) addressed the challenge of ensuring that a behavioral solution, derived in a source task could be meaningfully transferred to be a workable solution in a dissimilar and more complex target task. Hence, a mapping function is required so that learned behaviors are transferable between tasks with different states and state-action variables.…”

“…Hence, a mapping function is required so that learned behaviors are transferable between tasks with different states and state-action variables. Taylor et al (2010) derived the inter-task mappings for policy search method to transfer populations of control policies (ANN controllers) between the keep-away soccer, knight's joust, and server job scheduling tasks. This method was successfully applied with manually coded inter-task mapping functions as well as mapping functions that were only partially available or learned before behavior transfer.…”

“…Furthermore, related work intersecting the fields of evolutionary controller design and policy (behavior) transfer 1 indicate that coupling evolutionary search with the transfer of behaviors between tasks of increasing complexity is an effective means to boost evolved behavior quality for a broad range of tasks Taylor et al, 2010;Verbancsics and Stanley, 2010;Didi and Nitschke, 2016a). Policy transfer is a method that aims to improve learning by leveraging knowledge from learning in related but simpler tasks (Pan and Yang, 2010).…”