Multiagent Learning with a Noisy Global Reward Signal

Abstract

Scaling multiagent reinforcement learning to domains with many agents is a complex problem. In particular, multiagent credit assignment becomes a key issue as the system size increases. Some multiagent systems suffer from a global reward signal that is very noisy or difficult to analyze. This makes deriving a learnable local reward signal very difficult. Difference rewards (a particular instance of reward shaping) have been used to alleviate this concern, but they remain difficult to compute in many domains. In this paper we present an approach to modeling the global reward using function approximation that allows the quick computation of local rewards. We demonstrate how this model can result in significant improvements in behavior for three congestion problems: a multiagent ``bar problem'', a complex simulation of the United States airspace, and a generic air traffic domain. We show how the model of the global reward may be either learned on- or off-line using either linear functions or neural networks. For the bar problem, we show an increase in reward of nearly 200% over learning using the global reward directly. For the air traffic problem, we show a decrease in costs of 25% over learning using the global reward directly.

Learning and Exploiting Shaped Reward Models for Large Scale Multiagent RL

“…We use following baselines. Two domain specific approaches-AT-BASELINE (Brittain and Wei 2019) and AT-DR (Proper and Tumer 2013). LOCAL-DR (Colby et al 2016) is another DR based baseline.…”

“…We now show how to compute difference rewards (DRs) from r w in a model-free setting without requiring any extra simulations or domain expertise. Unlike previous approaches (Proper and Tumer 2013), our methods is not tied to a domain, and also explicitly utilizes aggregate information over all the agents, in contrast to methods that use only local information available to an agent (Colby et al 2016). As a result, our method results in much better solution quality when combined with a policy optimization technique than such previous approaches.…”

“…However, computing DRs is challenging as it requires access to the actual reward model (not available in a model free setting), or perform additional simulations, which is computationally challenging. To address this, previous approaches have used function approximation to learn the reward model (and DRs) from empirical returns (Proper and Tumer 2013;Colby et al 2016).…”

It’s All about Reward: Contrasting Joint Rewards and Individual Reward in Centralized Learning Decentralized Execution Algorithms

“…The closest works addressing team rewards in cooperative settings that we could find include works on difference rewards which try to measure the impact of an individual agent's actions on the full system reward [12]. The high learnability, among other nice properties, makes difference rewards attractive but impractical, due to the required knowledge of the total system state [13][14][15].…”