Decomposition methods with deep corrections for reinforcement learning

Abstract

Decomposition methods have been proposed to approximate solutions to large sequential decision making problems. In contexts where an agent interacts with multiple entities, utility decomposition can be used to separate the global objective into local tasks considering each individual entity independently. An arbitrator is then responsible for combining the individual utilities and selecting an action in real time to solve the global problem. Although these techniques can perform well empirically, they rely on strong assumptions of independence between the local tasks and sacrifice the optimality of the global solution. This paper proposes an approach that improves upon such approximate solutions by learning a correction term represented by a neural network. We demonstrate this approach on a fisheries management problem where multiple boats must coordinate to maximize their catch over time as well as on a pedestrian avoidance problem for autonomous driving. In each problem, decomposition methods can scale to multiple boats or pedestrians by using strategies involving one entity. We verify empirically that the proposed correction method significantly improves the decomposition method and outperforms a policy trained on the full scale problem without utility decomposition. decomposition methods with deep corrections for reinforcement learning 2 bined in real-time through an arbitrator that maximizes the expected utility [22]. Applications to aircraft collision avoidance with multiple intruders explore summing or using the minimum state-action values [7], [20]. While these approaches tend to perform well empirically, they sacrifice optimality. The solution to each subproblem assumes that its individual policy will be followed regardless of other entities, in contrast to the global policy considering all subproblems [23].Previous approaches to scale decision algorithms using decomposition methods relied on a distributed agent architecture [22]. Each agent is responsible for addressing one of the multiple objectives required to achieve a complex task. At each time step an arbitrator must decide between the different actions recommended by these agents and address possible conflicts. Possible arbitration strategies are command fusion [7], voting [22], lexicographic ordering [33] or utility fusion [22], [23]. It has been shown that utility fusion offers a more principled way of deciding between the individual agents compared to a voting-based approach or command fusion [22]. An alternative approach in leader follower scenarios consists of reducing the problem of controlling the group of agent to controlling a single group leader [15]. Other approaches rely on distributed learning algorithms, such as independent Q-learning, where each agent learns a policy without being aware of the other agents actions [28]. The underlying assumption of independence between agents in these algorithms trades off the benefit of collaboration for an easier learning of the task [8].Utility decomposition methods have been used in many practic...

Safe Reinforcement Learning for Automated Vehicles via Online Reachability Analysis

“…We assume that Γ is always drivable. Note that we denote the control input as u and the actions of the RL agent as a RL , which encodes trajectories of control inputs in our setting, as detailed later in (22). We use the projection operator proj (x) to map a state x ∈ X to its elements specified by .…”

Genetic-Algorithm-Aided Deep Reinforcement Learning for Multi-Agent Drone Delivery

“…These individual functions are then combined in real time to solve the overall problem while sacrificing optimality. Deep-neural-network-based correction methods are then applied to learn correction terms to improve the performance [55].…”

“…Each subtask is then solved separately in isolation. As solving the global problem suffers from the curse of dimensionality, solving each isolated subtask requires exceptionally less computation power [55]. At the end of the process each solution is combined through an approximate function f , also named the utility fusion function, such as:…”

“…According to [55], for the independent agents two approximation functions, max-sum and max-min, can be utilized to combine the subtask solutions:…”

“…This technique makes use of the decomposed utility functions to guide the RL agent toward an ideal policy while using less training data [55]. The architecture of the deep correction method is depicted in Figure 6.…”