Experience Replay Optimisation via ATSC and TSC for Performance Stability in Deep RL

Abstract

Catastrophic forgetting is a significant challenge in deep reinforcement learning (RL). To address this problem, researchers introduce the experience replay (ER) concept to complement the training of a deep RL agent. However, the buffer size, experience selection, and experience retention strategies adopted for the ER can negatively affect the agent’s performance stability, especially for complex continuous state action problems. This paper investigates how to address the stability problem using an enhanced ER method that combines a replay policy network, a dual memory, and an alternating transition selection control (ATSC) mechanism. Two frameworks were designed: an experience replay optimisation via alternating transition selection control (ERO-ATSC) without a transition storage control (TSC) and an ERO-ATSC with a TSC. The first is a hybrid of experience replay optimisation (ERO) and dual-memory experience replay (DER) and the second, which has two versions of its kind, integrates a transition storage control (TSC) into the first framework. After comprehensive experimental evaluations of the frameworks on the pendulum-v0 environment and across multiple buffer sizes, retention strategies, and sampling ratios, the reward version of ERO-ATSC with a TSC exhibits superior performance over the first framework and other novel methods, such as the deep deterministic policy gradient (DDPG) and ERO.

Explaining Deep Q-Learning Experience Replay with SHapley Additive exPlanations

“…However, its limited explainability, especially in Deep Reinforcement Learning (DRL), has hindered its adoption in regulated sectors like manufac-turing [2], finance [3], and health [4]. These challenges, including debugging, interpretation, and inefficiency, become more pronounced as Deep Learning advances, such as in improving Experience Replay [12][13][14][29][30][31][32]. While many have focused on improving Experience Replay's selection strategy, reducing its capacity remains largely unexplored.…”

“…However, drawbacks included correlated samples, limited capacity causing an agent to forget information, outdated samples from non-stationary environments, and overfitting from samples memorized. Prioritized Experience Replay (PER) [29], Attention-Based Experience Replay [30], Combined Experience Replay (CER) [12], Explanation-Aware Experience Replay [31] and many more selection strategies [12][13][14]32] propose solutions to these drawbacks. Understanding Experience Replay is crucial for efficiency.…”

“…As Figure 4 revealed the DCQL model was too simplistic to score high and some games proved too challenging to get any suitable results (Montezuma's Revenge and Venture). Experience Selection based on random uniform sampling is not an optimal selection strategy [12][13][14][29][30][31][32] and as Experience Replay is reduced there is a risk that rule dense features will be lost however capacity can be reduced from 1 × 10 4 to 5 × 10 3 without significantly affecting reward. This in turn will improve efficiency by reducing the impact on resources.…”

Improving Deep Deterministic Policy Gradient with Compact Experience Replay

“…Wei et al [43], propose a quantum-inspired ER paradigm to achieve a better balance between exploration and exploitation. Recent works address the problem of catastrophic forgetting by proposing strategies to improve control over the mechanisms for storing, selecting, retaining, and forgetting in ER [44], including using transfer learning about past experiences [45]. Li et al [46] proposed the Self-generated Long-term Experience Replay (SLER) method to mitigate catastrophic forgetting in continuous learning tasks.…”

Advances and Challenges in Learning from Experience Replay

“…It is mainly related to buffer size limitation and is sensitive to experience sampling and storage strategies. Recent works addressed this problem by proposing strategies to improve control over the mechanisms for storing, selecting, retaining, and forgetting in experience replay (Osei and Lopez, 2023), including using transfer learning about past experiences (Anzaldo and Andrade, 2022). According to , their Self-generated Long-term Experience Replay (SLER) approach improved the dual experience replay algorithm, applied in continuous learning tasks to mitigate catastrophic forgetting by reducing its impact on computer memory-consuming growth.…”