Explaining deep reinforcement learning decisions in complex multiagent settings: towards enabling automation in air traffic flow management

XDQN: Inherently Interpretable DQN through Mimicking

“…Given a minibatch of states, we calculate the MAE of this minibatch for any action as the mean absolute difference between the Q-values estimated by the mimic learner and the Q-values estimated by the deep Q-network for that action. More formally, for a minibatch of states 𝐷 𝑠 , the MAE 𝑖 of action 𝑎 𝑖 is denoted as: We focus on providing aggregated interpretations, focusing on the contribution of features to local decisions and to the overall policy: This, as suggested by ATM operators, is beneficial towards understanding decisions, helping them to increase their confidence to the solutions proposed, and mastering the inherent complexity in such a multi-agent setting, as solutions may be due to complex phenomena that are hard to be traced [15]. Specifically, in this work, local explainability measures state features' importance on a specific instance (i.e.…”

“…The tuple containing all agents' local states is the joint global state. Q-learning [33] agents has been shown to achieve remarkable performance on this task [15]. In our experiments, all agents share parameters and replay buffer and act independently.…”

“…Similarly, we consider local rewards and joint (global) rewards. The local reward is related to the cost per minute within a hotspot, the total duration of the flight (agent) in hotspots as well as to the delay that a flight has accumulated up to the simulation timestep [15].…”

“…Deep Reinforcement Learning (DRL) has mastered decision making policies in various difficult control tasks [11] [18] [15], games [22] [13] and other real-time applications [14] [37]. Despite the remarkable performance of DRL models, the knowledge of mastering these tasks remains implicit in deep neural networks.…”