Curriculum Learning for Cumulative Return Maximization

Abstract

Curriculum learning has been successfully used in reinforcement learning to accelerate the learning process, through knowledge transfer between tasks of increasing complexity. Critical tasks, in which suboptimal exploratory actions must be minimized, can benefit from curriculum learning, and its ability to shape exploration through transfer. We propose a task sequencing algorithm maximizing the cumulative return, that is, the return obtained by the agent across all the learning episodes. By maximizing the cumulative return, the agent not only aims at achieving high rewards as fast as possible, but also at doing so while limiting suboptimal actions. We experimentally compare our task sequencing algorithm to several popular metaheuristic algorithms for combinatorial optimization, and show that it achieves significantly better performance on the problem of cumulative return maximization. Furthermore, we validate our algorithm on a critical task, optimizing a home controller for a micro energy grid.

Curriculum Learning with a Progression Function

“…This section starts with the comparison between the Exponential progression and the Friction-Based progression. It then compares the performance of our approach with two other state-of-the-art Curriculum learning algorithms, as outlined in the Related Work section: Florensa et al [2] and Foglino et al [3].…”

“…The formulation of sequencing as a combinatorial optimization problem [4] over the intermediate tasks lends itself to the design of globally optimal sequencing algorithms. One such algorithm is Heuristic Task Sequencing for Cumulative Return [3] (HTS-CR), which is a complete anytime algorithm, converging to the optimal curriculum of a maximum length. Due to this guarantee of optimality, we use HTS-CR as one of the baselines to evaluate our sequencing method.…”

“…When using HTS-CR [3] as one of our baselines, we used transfer between tasks; in the case of the Grid World domain, the transfer method is the one outlined in the paper, on the other hand, on the Point Maze domain, the transfer method consisted in directly transferring the neural network from one task to the next. On both domains, the intermediate tasks used by HTS-CR were a subset of the tasks used by our approach; this gives some insight on the quality of the solution found by our approach compared to what is possible with a standard curriculum.…”

“…In order to show the potential of our framework, we compared the Frictionbased progression with other two state-of-the-art Curriculum Learning algorithms HTS-CR [3] and Reverse Curriculum Generation [2]. Figures 5 and 6 report the average and 95 percent confidence intervals on both test domains.…”

Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey

“…While metaheuristic algorithms are broadly applicable, it is also possible to create specific heuristic search methods targeted at particular problems, such as task sequencing with a specific transfer metric objective. Foglino et al (2019b) introduce one such heuristic search algorithm, designed to optimize for the cumulative return. Their approach begins by computing transferability between all pairs of tasks, using a simulator to estimate the cumulative return attained by using one task as a source for another.…”