Model-Based Quality-Diversity Search for Efficient Robot Learning

Keller, Leon; Tanneberg, Daniel; Stark, Svenja; Peters, Jan

doi:10.48550/arxiv.2008.04589

Cited by 4 publications

(5 citation statements)

References 19 publications

(31 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first approach is to leverage the efficiency of other optimization methods such as evolution strategies [7,11,19] and policy-gradients [40,43]. The other line of work known as model-based qualitydiversity [21,28,34], reduces the number of evaluations required 1 will be made available after acceptance. through the use of surrogate models to provide a prediction of the BD and fitness.…”

Section: Related Work 21 Quality-diversitymentioning

confidence: 99%

Accelerated Quality-Diversity for Robotics through Massive Parallelism

Lim¹,

Allard²,

Luca³

et al. 2022

Preprint

View full text Add to dashboard Cite

Quality-Diversity (QD) algorithms are a well-known approach to generate large collections of diverse and high-quality policies. However, QD algorithms are also known to be data-inefficient, requiring large amounts of computational resources and are slow when used in practice for robotics tasks. Policy evaluations are already commonly performed in parallel to speed up QD algorithms but have limited capabilities on a single machine as most physics simulators run on CPUs. With recent advances in simulators that run on accelerators, thousands of evaluations can performed in parallel on single GPU/TPU. In this paper, we present QDax, an implementation of MAP-Elites which leverages massive parallelism on accelerators to make QD algorithms more accessible. We first demonstrate the improvements on the number of evaluations per second that parallelism using accelerated simulators can offer. More importantly, we show that QD algorithms are ideal candidates and can scale with massive parallelism to be run at interactive timescales. The increase in parallelism does not significantly affect the performance of QD algorithms, while reducing experiment runtimes by two factors of magnitudes, turning days of computation into minutes. These results show that QD can now benefit from hardware acceleration, which contributed significantly to the bloom of deep learning.

show abstract

Section: Related Work 21 Quality-diversitymentioning

confidence: 99%

Accelerated Quality-Diversity for Robotics through Massive Parallelism

Lim¹,

Allard²,

Luca³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…These methods have been used to generate repertoires of primitive actions for robot locomotion [23], [16] or simple manipulations with a robot arm [17]. The generated repertoires have already been used to bootstrap deep learning approaches [25], [26]. We propose to extend novelty-based repertoire generation methods with exploration of multiple behavior spaces such as in [27], [28] but with an additional focus on setups with very sparse interactions.…”

Section: Related Workmentioning

confidence: 99%

Automatic Acquisition of a Repertoire of Diverse Grasping Trajectories through Behavior Shaping and Novelty Search

Morel¹,

Kunimoto²,

Coninx³

et al. 2022

Preprint

View full text Add to dashboard Cite

Grasping a particular object may require a dedicated grasping movement that may also be specific to the robot end-effector. No generic and autonomous method does exist to generate these movements without making hypotheses on the robot or on the object. Learning methods could help to autonomously discover relevant grasping movements, but they face an important issue: grasping movements are so rare that a learning method based on exploration has little chance to ever observe an interesting movement, thus creating a bootstrap issue. We introduce an approach to generate diverse grasping movements in order to solve this problem. The movements are generated in simulation, for particular object positions. We test it on several simulated robots: Baxter, Pepper and a Kuka Iiwa arm. Although we show that generated movements actually work on a real Baxter robot, the aim is to use this method to create a large dataset to bootstrap deep learning methods.

show abstract

“…SAIL integrates surrogate models, in the form of Gaussian Process (GP) models, to approximate the objective function and reduce the number of evaluations for the computationally expensive application of aerodynamic design. Another algorithm called M-QD [18] later follows up on this idea and used neural network models that map the parameter space to the behaviour and fitness space as a surrogate model. They demonstrate this on robotic pushing and placing tasks.…”

Section: Model-based Quality-diversitymentioning

confidence: 99%

Learning to Walk Autonomously via Reset-Free Quality-Diversity

Lim,

Reichenbach,

Cully

2022

Preprint

View full text Add to dashboard Cite

Quality-Diversity (QD) algorithms can discover large and complex behavioural repertoires consisting of both diverse and highperforming skills. However, the generation of behavioural repertoires has mainly been limited to simulation environments instead of real-world learning. This is because existing QD algorithms need large numbers of evaluations as well as episodic resets, which require manual human supervision and interventions. This paper proposes Reset-Free Quality-Diversity optimization (RF-QD) as a step towards autonomous learning for robotics in open-ended environments. We build on Dynamics-Aware Quality-Diversity (DA-QD) and introduce a behaviour selection policy that leverages the diversity of the imagined repertoire and environmental information to intelligently select of behaviours that can act as automatic resets. We demonstrate this through a task of learning to walk within defined training zones with obstacles. Our experiments show that we can learn full repertoires of legged locomotion controllers autonomously without manual resets with high sample efficiency in spite of harsh safety constraints. Finally, using an ablation of different target objectives, we show that it is important for RF-QD to have diverse types solutions available for the behaviour selection policy over solutions optimised with a specific objective. Videos and code available at https://sites.google.com/view/rf-qd.

show abstract

Model-Based Quality-Diversity Search for Efficient Robot Learning

Cited by 4 publications

References 19 publications

Accelerated Quality-Diversity for Robotics through Massive Parallelism

Accelerated Quality-Diversity for Robotics through Massive Parallelism

Automatic Acquisition of a Repertoire of Diverse Grasping Trajectories through Behavior Shaping and Novelty Search

Learning to Walk Autonomously via Reset-Free Quality-Diversity

Contact Info

Product

Resources

About