Agent Environment Cycle Games

Abstract

Partially Observable Stochastic Games (POSGs), are the most general model of games used in Multi-Agent Reinforcement Learning (MARL), modeling actions and observations as happening sequentially for all agents. We introduce Agent Environment Cycle Games (AEC Games), a model of games based on sequential agent actions and observations. AEC Games can be thought of as sequential versions of POSGs, and we prove that they are equally powerful. We argue conceptually and through case studies that the AEC games model is useful in important scenarios in MARL for which the POSG model is not well suited. We additionally introduce "cyclically expansive curriculum learning," a new MARL curriculum learning method motivated by the AEC games model. It can be applied "for free, " and experimentally we show this technique to achieve up to 35.1% more total reward on average.

Deep Reinforcement Learning Multi-Agent System for Resource Allocation in Industrial Internet of Things

“…The mathematical model that considers a sequential-instead of simultaneous-decisionmaking of the agents is the Agent Environment Cycle (AEC) game [39]. In [37], Terry et al prove that, for every POMG, an equivalent AEC game exists and vice versa.…”

“…It is to differentiate between two cases for status update from s t to s t+1 . For environment steps, (i = 0) applies that the next state s t+1 is random and occurs with the probability of the transition function P. Otherwise, for agent steps i > 0, a deterministic state transition according to the transition function T i takes place [39]. Afterwards, the next agent i with the probability of the next-agent function v(i |s t , i, a i t ) is chosen.…”