Quantifying the effects of environment and population diversity in multi-agent reinforcement learning

Abstract

Generalization is a major challenge for multi-agent reinforcement learning. How well does an agent perform when placed in novel environments and in interactions with new co-players? In this paper, we investigate and quantify the relationship between generalization and diversity in the multi-agent domain. Across the range of multi-agent environments considered here, procedurally generating training levels significantly improves agent performance on held-out levels. However, agent performance on the specific levels used in training sometimes declines as a result. To better understand the effects of co-player variation, our experiments introduce a new environment-agnostic measure of behavioral diversity. Results demonstrate that population size and intrinsic motivation are both effective methods of generating greater population diversity. In turn, training with a diverse set of co-players strengthens agent performance in some (but not all) cases.

A Survey on Population-Based Deep Reinforcement Learning

“…Without adequate diversity, the population may converge prematurely to suboptimal solutions, leading to the stagnation of the learning process. Overfitting to policies in the policy zoo is a significant challenge to generalization [25,59]. Although the diversity of a population has been widely discussed in the evolutionary algorithm community at the genotype level, phenotype level, and the combination of the previous two cases [60], which typically operate on a fixed set of candidate solutions, PB-DRL is often used in dynamic and uncertain environments where the population size and diversity can change over time.…”

Diversity for transfer in learning-based control of buildings

“…This problem is exacerbated in MARL as agents tend to overfit to their co-players [23]. A interesting approach to address this problem is to incorporate diversity when learning a control policy [28]. Diversity approaches can be broadly classified into two categories: environmental diversity [19,28] and policy diversity [12,25,26].…”

“…A interesting approach to address this problem is to incorporate diversity when learning a control policy [28]. Diversity approaches can be broadly classified into two categories: environmental diversity [19,28] and policy diversity [12,25,26]. In environmental diversity, variants of the given environment are used to train the agents.…”

“…In environmental diversity, variants of the given environment are used to train the agents. The goal is to obtain policies that capture main features of the environment, hence generalize to a broader class of environments [19,28]. Policy diversity focuses on finding distinct sub-optimal policies in the training environment.…”

“…Since the training and target buildings might differ with respect to their structure, floor plan, HVAC system, and occupancy pattern, the near-optimal policy found in the training building can perform poorly on the target building. Previous work that studies the relationship between generalization and diversity in RL (more detail in Section 2.3) suggests that agents trained with diversity exhibit stronger generalization to novel environments [28]. Inspired by this, in this paper we cast HVAC control as a multi-agent reinforcement learning problem where diversity is incorporated in the training process of each agent.…”