Diversity policy gradient for sample efficient quality-diversity optimization

Pierrot, Thomas; Macé, Valentin; Chalumeau, Félix; Flajolet, Arthur; Cideron, Geoffrey; Beguir, Karim; Cully, Antoine; Sigaud, Olivier; Perrin-Gilbert, Nicolas

doi:10.1145/3512290.3528845

Cited by 29 publications

(13 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, there are fewer algorithms oriented towards structured action spaces in general, and in the tasks solved by these algorithms, there are no explicit dependencies between atomic actions. Thus, existing approaches are either based on the assumption of independence of the decomposed sub-actions [51,78]; or they are based on the inductive bias to assign a conditional dependency structure to the decomposed sub-actions and pick up the actions one by one through an autoregressive form based on recurrent neural networks, which are finally spliced into the original actions [58,68]. There are also a series of approaches that assume a game relationship between the decomposed sub-actions, model each sub-action as an agent, and use MARL methods to solve them [21,42,52,83,93].…”

Section: A2 Structured or Large-scale Actionsmentioning

confidence: 99%

Diverse Policy Optimization for Structured Action Space

Li¹,

Wang²,

Yang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Enhancing the diversity of policies is beneficial for robustness, exploration, and transfer in reinforcement learning (RL). In this paper, we aim to seek diverse policies in an under-explored setting, namely RL tasks with structured action spaces with the two properties of composability and local dependencies. The complex action structure, non-uniform reward landscape, and subtle hyperparameter tuning due to the properties of structured actions prevent existing approaches from scaling well. We propose a simple and effective RL method, Diverse Policy Optimization (DPO), to model the policies in structured action space as the energy-based models (EBM) by following the probabilistic RL framework. A recently proposed novel and powerful generative model, GFlowNet, is introduced as the efficient, diverse EBM-based policy sampler. DPO follows a joint optimization framework: the outer layer uses the diverse policies sampled by the GFlowNet to update the EBM-based policies, which supports the GFlowNet training in the inner layer. Experiments on ATSC and Battle benchmarks demonstrate that DPO can efficiently discover surprisingly diverse policies in challenging scenarios and substantially outperform existing state-of-the-art methods.

show abstract

Section: A2 Structured or Large-scale Actionsmentioning

confidence: 99%

Diverse Policy Optimization for Structured Action Space

Li¹,

Wang²,

Yang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Policy Gradient Assisted MAP-Elites (PGA-ME) [13]. Since MAP-Elites performs poorly when directly optimizing modern RL controllers [11], [13], [12], several algorithms [11], [13], [12], [39] have scaled it to such controllers. One such algorithm, PGA-ME, replaces the Gaussian noise mutation with two types of operations: (1) gradient ascent, performed with the TD3 algorithm [40], and (2) crossover, performed with a genetic algorithm [1].…”

Section: Background a Map-elitesmentioning

confidence: 99%

“…We consider algorithms in the MAP-Elites family which address the QD-RL problem formulated in Sec. II without making any assumptions on the measures, as opposed to algorithms based on novelty search[54] or algorithms which define measures similarly to an RL reward function[39].…”

mentioning

confidence: 99%

Training Diverse High-Dimensional Controllers by Scaling Covariance Matrix Adaptation MAP-Annealing

Tjanaka¹,

Fontaine²,

Aniruddha³

et al. 2022

Preprint

View full text Add to dashboard Cite

Pre-training a diverse set of robot controllers in simulation has enabled robots to adapt online to damage in robot locomotion tasks. However, finding diverse, highperforming controllers requires specialized hardware and extensive tuning of a large number of hyperparameters. On the other hand, the Covariance Matrix Adaptation MAP-Annealing algorithm, an evolution strategies (ES)-based quality diversity algorithm, does not have these limitations and has been shown to achieve state-of-the-art performance in standard benchmark domains. However, CMA-MAE cannot scale to modern neural network controllers due to its quadratic complexity. We leverage efficient approximation methods in ES to propose three new CMA-MAE variants that scale to very high dimensions. Our experiments show that the variants outperform ES-based baselines in benchmark robotic locomotion tasks, while being comparable with state-of-the-art deep reinforcement learningbased quality diversity algorithms. Source code and videos are available at https://scalingcmamae.github.io Compute archive improvement Adapt and with ES Sample solutions Evaluate policies and insert into archive

show abstract

“…QD in particular has been applied to a wide range of domains ranging from robotics [7], [8] to content generation [5], [9] or design [4]. Recently, many QD works have shifted focus toward Neuroevolution and the evolution of large closed-loop controllers for robotics [10]- [13].…”

Section: Introductionmentioning

confidence: 99%

Empirical analysis of PGA-MAP-Elites for Neuroevolution in Uncertain Domains

Flageat

Chalumeau

Cully

2023

ACM Trans. Evol. Learn. Optim.

Self Cite

View full text Add to dashboard Cite

Quality-Diversity algorithms, among which MAP-Elites, have emerged as powerful alternatives to performance-only optimisation approaches as they enable generating collections of diverse and high-performing solutions to an optimisation problem. However, they are often limited to low-dimensional search spaces and deterministic environments. The recently introduced Policy Gradient Assisted MAP-Elites (PGA-MAP-Elites) algorithm overcomes this limitation by pairing the traditional Genetic operator of MAP-Elites with a gradient-based operator inspired by Deep Reinforcement Learning. This new operator guides mutations toward high-performing solutions using policy-gradients. In this work, we propose an in-depth study of PGA-MAP-Elites. We demonstrate the benefits of policy-gradients on the performance of the algorithm and the reproducibility of the generated solutions when considering uncertain domains. We first prove that PGA-MAP-Elites is highly performant in both deterministic and uncertain high-dimensional environments, decorrelating the two challenges it tackles. Secondly, we show that in addition to outperforming all the considered baselines, the collections of solutions generated by PGA-MAP-Elites are highly reproducible in uncertain environments, approaching the reproducibility of solutions found by Quality-Diversity approaches built specifically for uncertain applications. Finally, we propose an ablation and in-depth analysis of the dynamic of the policy-gradients-based variation. We demonstrate that the policy-gradient variation operator is determinant to guarantee the performance of PGA-MAP-Elites but is only essential during the early stage of the process, where it finds high-performing regions of the search space.

show abstract

Diversity policy gradient for sample efficient quality-diversity optimization

Cited by 29 publications

References 11 publications

Diverse Policy Optimization for Structured Action Space

Diverse Policy Optimization for Structured Action Space

Training Diverse High-Dimensional Controllers by Scaling Covariance Matrix Adaptation MAP-Annealing

Empirical analysis of PGA-MAP-Elites for Neuroevolution in Uncertain Domains

Contact Info

Product

Resources

About