A Deep Multi-Agent Reinforcement Learning Approach to Autonomous Separation Assurance

Abstract

A novel deep multi-agent reinforcement learning framework is proposed to identify and resolve conflicts among a variable number of aircraft in a high-density, stochastic, and dynamic sector in en route airspace. Currently the sector capacity is limited by human air traffic controller's cognitive limitation. In order to scale up to a high-density airspace, in this work we investigate the feasibility of a new concept (autonomous separation assurance) and a new approach (multi-agent reinforcement learning) to push the sector capacity above human cognitive limitation. We propose the concept of using distributed vehicle autonomy to ensure separation, instead of a centralized sector air traffic controller. Our proposed framework utilizes an actor-critic model, Proximal Policy Optimization (PPO) that we customize to incorporate an attention network. By using the attention network, we are able to encode the information from a variable number of intruder aircraft into a fixed length vector and allow the agents to learn which intruder aircraft's information is critical to achieve the optimal performance. This allows the agents to have access to variable aircraft information in the sector in a scalable, efficient approach to achieve high traffic throughput under uncertainty. The agents are trained using a centralized learning, decentralized execution scheme where one neural network is learned and shared by all agents in the environment. To validate the proposed framework, we designed three challenging case studies in the BlueSky air traffic control environment. Numerical results show the proposed framework significantly reduces the offline training time without sacrificing performance.

Multi-UAV Conflict Resolution with Graph Convolutional Reinforcement Learning

“…[27], the authors use the Deep Deterministic Policy Gradient (DDPG) technique to mitigate conflicts in high density scenarios and uncertainties. Brittain et al [28] used a deep multiagent reinforcement learning framework to ensure autonomous separation between aircraft. Dalmau et al [29] used Message Passing Neural Networks (MPNN) to model air traffic control as a multiagent reinforcement learning system where agents must ensure conflict free flight through a sector.…”

Obstacle Avoidance for UAS in Continuous Action Space Using Deep Reinforcement Learning

“…The performance of the DQN algorithm in avoiding single aircraft to multiple aircraft is investigated in [34]. A novel deep multi-agent reinforcement learning framework based on PPO is proposed in [35] to detect and avoid conflicts among multiple aircraft in a highdensity and dynamic sector under uncertainty. The DRL work mentioned above is in continuous state and discrete action space.…”

Obstacle Avoidance for UAS in Continuous Action Space Using Deep Reinforcement Learning

“…The per-formance of the agent in avoiding single up to multiple aircraft by using the DQN algorithm is investigated in Keong et al (2019). Brittain et al (2020) proposed a novel deep multi-agent reinforcement learning framework based on PPO to identify and resolve conflicts among a variable number of aircraft in a high-density, stochastic, and dynamic sector in en-route airspace. The DRL work mentioned above is in continuous state and discrete action space.…”