Continuous Transition: Improving Sample Efficiency for Continuous Control Problems via MixUp

Abstract

Although deep reinforcement learning (RL) has been successfully applied to a variety of robotic control tasks, it's still challenging to apply it to real-world tasks, due to the poor sample efficiency. Attempting to overcome this shortcoming, several works focus on reusing the collected trajectory data during the training by decomposing them into a set of policyirrelevant discrete transitions. However, their improvements are somewhat marginal since i) the amount of the transitions is usually small, and ii) the value assignment only happens in the joint states. To address these issues, this paper introduces a concise yet powerful method to construct Continuous Transition, which exploits the trajectory information by exploiting the potential transitions along the trajectory. Specifically, we propose to synthesize new transitions for training by linearly interpolating the conjunctive transitions. To keep the constructed transitions authentic, we also develop a discriminator to guide the construction process automatically. Extensive experiments demonstrate that our proposed method achieves a significant improvement in sample efficiency on various complex continuous robotic control problems in MuJoCo and outperforms the advanced modelbased / model-free RL methods.

Neighborhood Mixup Experience Replay: Local Convex Interpolation for Improved Sample Efficiency in Continuous Control Tasks

“…Several reinforcement learning methodologies make use of Mixup-interpolated experiences for training reinforcement learning agents. In Continuous Transition (Lin et al 2020), temporally-adjacent transitions are interpolated with Mixup, generating synthetic transitions between pairs of consecutive transitions. In MixReg (Wang et al 2020), generated transitions are formed using Mixup on combinations of input and output signals.…”

“…While these approaches increase the training domain via interpolation, they do not strictly enforce geometric transition proximity of the resulting samples. Proximity between the points used for sampling is encoded temporally, as in (Lin et al 2020;Sinha, Mandlekar, and Garg 2021), but not in the geometric transition space of the agent's experience. NMER employs a nearest neighbor heuristic to encourage transition pairs for Mixup to be located approximately within the same dynamics regimes in the transition manifold.…”

“…NMER employs a nearest neighbor heuristic to encourage transition pairs for Mixup to be located approximately within the same dynamics regimes in the transition manifold. Compared to Continuous Transition (Lin et al 2020) and S4RL (Sinha, Mandlekar, and Garg 2021), samples interpolated with NMER may better preserve the local dynamics of the environment and enable further agent regularization through inter-episode interpolation between transitions and their dynamic sets of nearest neighbors.…”

“…Implementation details and ablation studies for each replay buffer variant are provided in the technical report. We measure replay buffer sample efficiency using the evaluation reward of the reinforcement learning agent after 200K environment interactions have been sampled, as in (Lin et al 2020;K. Lee et al 2020).…”

“…We compare NMER to the following baselines: (i) Uniform, Vanilla Replay (U) (Mnih et al 2013;Engel, Mannor, and Meir 2005), where transitions are sampled i.i.d. uniformly from the replay buffer, (ii) Prioritized Experience Replay (PER) (Schaul et al 2016) with stochastic prioritization, (iii) Continuous Transition (CT) (Lin et al 2020). Since the main comparison between NMER and CT is how samples are selected for interpolation, we make two modifications to the original CT baseline: (a) We remove the automatic Mixup α hyperparameter tuning mechanism, and (b) If a terminal state is encountered in either the sample or neighbor transition, no interpolation occurs, and the sampled transition is simply used for training the agent.…”