Decentralized Control of Quadrotor Swarms with End-to-end Deep Reinforcement Learning

Abstract

We demonstrate the possibility of learning drone swarm controllers that are zero-shot transferable to real quadrotors via large-scale multi-agent end-to-end reinforcement learning. We train policies parameterized by neural networks that are capable of controlling individual drones in a swarm in a fully decentralized manner. Our policies, trained in simulated environments with realistic quadrotor physics, demonstrate advanced flocking behaviors, perform aggressive maneuvers in tight formations while avoiding collisions with each other, break and re-establish formations to avoid collisions with moving obstacles, and efficiently coordinate in pursuit-evasion tasks. We analyze, in simulation, how different model architectures and parameters of the training regime influence the final performance of neural swarms. We demonstrate the successful deployment of the model learned in simulation to highly resource-constrained physical quadrotors performing station keeping and goal swapping behaviors. Code and video demonstrations are available at the project website https://sites.google.com/view/swarm-rl.

Deep Learning-Based Consensus Control of a Multi-Agents System with Unknown Time-Varying Delay

“…A neural network is exploited to approximate the value functions with which a Nash equilibrium on the policies of agents can be found for the consensus of MAS with nonlinear dynamics without model knowledge [31]. Eight quadrotors built on an open-source platform and a very small deep learning network were experimentally shown to achieve movements with maintaining formation [32].…”

“…Even though RL is an intriguing approach to achieve consensus without model knowledge, it has drawbacks such as a dependency on initial value [32] and convergence withunmeaningful policies [33]. In addition, extending RL from a single agent to multi-agents lays down some challenges such as the heterogeneity of agents, definition of a global goal, knowledge sharing, and the scalability of the number of agents [33].…”