Multiagent Cooperation and Competition with Deep Reinforcement Learning

Tampuu, Ardi; Matiisen, Tambet; Kodelja, Dorian; Kuzovkin, Ilya; Korjus, Kristjan; Aru, Juhan; Aru, Jaan; Vicente, Raúl

doi:10.48550/arxiv.1511.08779

Cited by 16 publications

(26 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To empirically support our theoretical results on EMGs, this section is devoted to introduce two simple decentralized extensions of singleagent Deep Reinforcement Learning (DRL) algorithms designed to work with LT L specifications. The first approach is based on extending with temporal logic specifications a popular baseline in MARL called I-DQN [30], while the second is a multi-agent extension of LPOPL that we referred in Sec 1. The extended algorithms described below are employed in the experiments presented in Sec.…”

Section: Deep Marl With Co-safe Lt L Goalsmentioning

confidence: 99%

“…This allows Independent Q-learning [32] to train multiple agents in a decentralized fashion. Here we consider a deep learning variant of this algorithm (see, e.g., [30]), where each agent is trained with an independent DQN. However, in our case, we adopt a decentralized version of an algorithm that uses LT L specifications and LT L progression instead of classical reward functions (see, e.g., [20]).…”

Section: I-dqn With Co-safe Lt L Goalsmentioning

confidence: 99%

“…Despite the growing success of MARL, also due to the adoption of neural networks in Deep (RL) [30], there is still little work on combining these multi-agent learning techniques with formal methods. This is the long-term challenge that we here start to address.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learning

León,

Belardinelli

2020

Preprint

View full text Add to dashboard Cite

The combination of Formal Methods with Reinforcement Learning (RL) has recently attracted interest as a way for singleagent RL to learn multiple-task specifications. In this paper we extend this convergence to multi-agent settings and formally define Extended Markov Games as a general mathematical model that allows multiple RL agents to concurrently learn various non-Markovian specifications. To introduce this new model we provide formal definitions and proofs as well as empirical tests of RL algorithms running on this framework. Specifically, we use our model to train two different logic-based multi-agent RL algorithms to solve diverse settings of non-Markovian co-safe LT L specifications.

show abstract

Section: Deep Marl With Co-safe Lt L Goalsmentioning

confidence: 99%

Section: I-dqn With Co-safe Lt L Goalsmentioning

confidence: 99%

See 1 more Smart Citation

Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learning

León,

Belardinelli

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Cooperative Cooperative CAD environments help in developing agent algorithms that can learn near-globally optimal policies for all the driving agents that act as a cooperative unit. Such environments help in developing agents that learn to communicate [9] and benefit from learning to cooperate [25]. This type of environments will enable development of efficient fleet of vehicles that cooperate and communicate with each other to reduce congestion, eliminate collisions and optimized traffic flows.…”

Section: Nature Of Tasksmentioning

confidence: 99%

“…Independent DQN [25] extends DQN to cooperative, fully-observable Multi-Agent setting, applied to a two-player pong environment, in which all agents independently learn and update their own Q-function Q i (s, a i ; θ i ).…”

Section: A Appendixamentioning

confidence: 99%

Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning

Palanisamy¹

2019

Preprint

View full text Add to dashboard Cite

The capability to learn and adapt to changes in the driving environment is crucial for developing autonomous driving systems that are scalable beyond geo-fenced operational design domains. Deep Reinforcement Learning (RL) provides a promising and scalable framework for developing adaptive learning based solutions. Deep RL methods usually model the problem as a (Partially Observable) Markov Decision Process in which an agent acts in a stationary environment to learn an optimal behavior policy. However, driving involves complex interaction between multiple, intelligent (artificial or human) agents in a highly non-stationary environment. In this paper, we propose the use of Partially Observable Markov Games(POSG) for formulating the connected autonomous driving problems with realistic assumptions. We provide a taxonomy of multi-agent learning environments based on the nature of tasks, nature of agents and the nature of the environment to help in categorizing various autonomous driving problems that can be addressed under the proposed formulation. As our main contributions, we provide MACAD-Gym, a Multi-Agent Connected, Autonomous Driving agent learning platform for furthering research in this direction. Our MACAD-Gym platform provides an extensible set of Connected Autonomous Driving (CAD) simulation environments that enable the research and development of Deep RL-based integrated sensing, perception, planning and control algorithms for CAD systems with unlimited operational design domain under realistic, multi-agent settings. We also share the MACAD-Agents that were trained successfully using the MACAD-Gym platform to learn control policies for multiple vehicle agents in a partially observable, stop-sign controlled, 3-way urban intersection environment with raw (camera) sensor observations.

show abstract

Towards Interpretable Policies in Multi-agent Reinforcement Learning Tasks

Crespi

Custode²,

Iacca³

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Deep Learning (DL) allowed the field of Multi-Agent Reinforcement Learning (MARL) to make significant advances, speeding-up the progress in the field. However, agents trained by means of DL in MARL settings have an important drawback: their policies are extremely hard to interpret, not only at the individual agent level, but also (and especially) considering the fact that one has to take into account the interactions across the whole set of agents. In this work, we make a step towards achieving interpretability in MARL tasks. To do that, we present an approach that combines evolutionary computation (i.e., grammatical evolution) and reinforcement learning (Q-learning), which allows us to produce agents that are, at least to some extent, understandable. Moreover, differently from the typically centralized DL-based approaches (and because of the possibility to use a replay buffer), in our method we can easily employ Independent Q-learning to train a team of agents, which facilitates robustness and scalability. By evaluating our approach on the Battlefield task from the MAgent implementation in the PettingZoo library, we observe that the evolved team of agents is able to coordinate its actions in a distributed fashion, solving the task in an effective way.

show abstract

Multiagent Cooperation and Competition with Deep Reinforcement Learning

Cited by 16 publications

References 0 publications

Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learning

Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learning

Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning

Towards Interpretable Policies in Multi-agent Reinforcement Learning Tasks

Contact Info

Product

Resources

About