Batch Prioritization in Multigoal Reinforcement Learning

Abstract

In multigoal reinforcement learning, an agent interacts with an environment and learns to achieve multiple goals. The goal-conditioned policy is trained to effectively generalize its behavior for multiple goals. During training, the experiences collected by the agent are randomly sampled from a replay buffer. Because biased sampling of achieved goals affects the success rate of a given task, it should be avoided by considering the valid goal space, introduced here as the set of goals to achieve, and the current competence of the policy. To this end, a novel prioritization method for creation of batches, e.g., collections of samples, is proposed. Candidate batches are sampled and associated with costs; in each iteration the batch with the minimum cost is chosen to train the policy. The cost function is modeled by an intended goal, which is proposed as a hypothetical goal that the policy is trying to learn in each cycle, and the information of the valid goal space. The minimum cost of the batch selected for each iteration decreases throughout training as the policy learns to achieve goals near the center of the valid goal space. The proposed batch prioritization method is combined with hindsight experience replay (HER) for experiments in robotic control tasks presented in the OpenAI gym suite to demonstrate learning performance comparable to that of other state-of-the-art prioritization methods. As a result, the proposed batch prioritization method can achieve improved learning performance in 4 out of 5 tasks, particularly for harder tasks. The experimental results suggest that the proposed method for the creation of training batches, using the valid goal space information and current competence of the policy, can enhance learning performance in multigoal tasks with high-dimensional goal space.

Cluster-Based Sampling in Hindsight Experience Replay for Robotic Tasks (Student Abstract)

“…The three tasks are Push, PickAndPlace, and Slide tasks. The integration of the proposed sampling strategy into HER is based on the implementation in (Vecchietti et al 2020).…”

Enhanced Off-Policy Reinforcement Learning With Focused Experience Replay

“…In practice, older experiences often originate from unsuccessful episodes, whereas recent experiences are often from successful episodes and thus have more useful information. For instance, unsuccessful episodes in the initial stage mostly contain environment steps with inappropriate action choices [21] that do not provide useful information. However, an RL agent should not aggressively utilize newer experiences and ignore the older ones, which causes the agent's failure to generalize the knowledge and to experience a catastrophic forgetting [22].…”

Two-stage training algorithm for AI robot soccer

“…Recently, deep reinforcement learning (DRL) has been widely applied to deterministic games ( Silver et al, 2018 ), video games ( Mnih et al, 2015 ; Mnih et al, 2016 ; Silver et al, 2016 ), sensor networks ( Kim et al, 2020 ), and complex robotic tasks ( Andrychowicz et al, 2017 ; Hwangbo et al, 2019 ; Seo et al, 2019 ; Vecchietti et al, 2020 ; Vecchietti, Seo & Har, 2020 ). Despite the breakthrough results achieved in the field of DRL, deep learning in multi-agent environments that require both cooperation and competition is still challenging.…”