Automatic Curriculum Graph Generation for Reinforcement Learning Agents

Abstract

In recent years, research has shown that transfer learning methods can be leveraged to construct curricula that sequence a series of simpler tasks such that performance on a final target task is improved. A major limitation of existing approaches is that such curricula are handcrafted by humans that are typically domain experts. To address this limitation, we introduce a method to generate a curriculum based on task descriptors and a novel metric of transfer potential. Our method automatically generates a curriculum as a directed acyclic graph (as opposed to a linear sequence as done in existing work). Experiments in both discrete and continuous domains show that our method produces curricula that improve the agent's learning performance when compared to the baseline condition of learning on the target task from scratch.

A Versatile Adaptive Curriculum Learning Framework for Task-oriented Dialogue Policy Learning

“…But current DRL-based dialogue policy approaches mostly remain brute-force random sampling training, improving their performance at the expense of high interaction costs (Jiang et al, 2015;Ren et al, 2018;Narvekar and Stone, 2019;Narvekar et al, 2020). Inspired by human education, a novel training paradigm, curriculum learning (CL), is proposed to improve learning performance and efficiency through training a model on a designed sequence of training tasks, rather than an arbitrary random sampling (Svetlik et al, 2017;Fan et al, 2018;Racanière et al, 2019;Green et al, 2019). Although many empirical studies demonstrated beneficial effects of CL, reporting in the field of dialogue policy remains very limited (Zhao et al, 2021a;Liu et al, 2021).…”

Subtask-masked curriculum learning for reinforcement learning with application to UAV maneuver decision-making

“…We may also consider more automatic sequencing strategies, such as using a subtask set to calculate transfer potential (Svetlik et al, 2017;Silva and Costa, 2018). However, we leave it as future work and focus on how to transfer the knowledge learned from intermediate tasks to improve performance in the target task.…”

Curriculum Generation and Sequencing for Deep Reinforcement Learning in StarCraft II

“…Depending on the problem itself, various domain-specific measures could also be used for difficulty evaluation [12], further complicating the issue. Furthermore, all existing research efforts on generating tasks for CL in RL have focused on attempting to automate the process and all involve some degree of limitation [22].…”