Autonomous agents modelling other agents: A comprehensive survey and open problems

Albrecht, Stefano V.; Stone, Peter

doi:10.1016/j.artint.2018.01.002

Cited by 300 publications

(217 citation statements)

References 163 publications

(261 reference statements)

Supporting

Mentioning

216

Contrasting

Unclassified

Order By: Relevance

“…Our goal is to outline a recent and active area (i.e., MDRL), as well as to motivate future research to take advantage of the ample and existing literature in multiagent learning. We aim to enable researchers with experience in either DRL or MAL to gain a common understanding about recent works, and open problems in MDRL, and to avoid having scattered sub-communities with little interaction [2,10,11,38].…”

Section: Introductionmentioning

confidence: 99%

“…Despite this complexity, top AI conferences like AAAI, ICML, ICLR, IJCAI and NeurIPS, and specialized conferences such as AAMAS, have published works reporting successes in MDRL. In light of these works, we believe it is pertinent to first, have an overview of the recent MDRL works, and second, understand how these recent works relate to the existing literature.This article contributes to the state of the art with a brief survey of the current works in MDRL in an effort to complement existing surveys on multiagent learning [36,10], cooperative learning [7,8], agents modeling agents [11], knowledge reuse in multiagent RL [12], and (singleagent) deep reinforcement learning [23,37].First, we provide a short review of key algorithms in RL such as Q-learning and REINFORCE (see Section 2.1). Second, we review DRL highlighting the challenges in this setting and reviewing recent works (see Section 2.2).…”

mentioning

confidence: 99%

See 1 more Smart Citation

A survey and critique of multiagent deep reinforcement learning

Hernández-Leal¹,

Kartal²,

Taylor³

2019

Auton Agent Multi-Agent Syst

439

240

View full text Add to dashboard Cite

Deep reinforcement learning (RL) has achieved outstanding results in recent years. This has led to a dramatic increase in the number of applications and methods. Recent works have explored learning beyond single-agent scenarios and have considered multiagent learning (MAL) scenarios. Initial results report successes in complex multiagent domains, although there are several challenges to be addressed. The primary goal of this article is to provide a clear overview of current multiagent deep reinforcement learning (MDRL) literature. Additionally, we complement the overview with a broader analysis: (i) we revisit previous key components, originally presented in MAL and RL, and highlight how they have been adapted to multiagent deep reinforcement learning settings. (ii) We provide general guidelines to new practitioners in the area: describing lessons learned from MDRL works, pointing to recent benchmarks, and outlining open avenues of research. (iii) We take a more critical tone raising practical challenges of MDRL (e.g., implementation and computational demands). We expect this article will help unify and motivate future research to take advantage of the abundant literature that exists (e.g., RL and MAL) in a joint effort to promote fruitful research in the multiagent community.$ Earlier versions of this work had the title: "Is multiagent deep reinforcement learning the answer or the question? A brief survey" arXiv:1810.05587v3 [cs.MA] 30 Aug 2019 Go [14,15], poker [16,17], and games of two competing teams, e.g., DOTA 2 [18] and StarCraft II [19].While different techniques and algorithms were used in the above scenarios, in general, they are all a combination of techniques from two main areas: reinforcement learning (RL) [20] and deep learning [21,22].RL is an area of machine learning where an agent learns by interacting (i.e., taking actions) within a dynamic environment. However, one of the main challenges to RL, and traditional machine learning in general, is the need for manually designing quality features on which to learn. Deep learning enables efficient representation learning, thus allowing the automatic discovery of features [21,22]. In recent years, deep learning has had successes in different areas such as computer vision and natural language processing [21,22]. One of the key aspects of deep learning is the use of neural networks (NNs) that can find compact representations in high-dimensional data [23].In deep reinforcement learning (DRL) [23,24] deep neural networks are trained to approximate the optimal policy and/or the value function. In this way the deep NN, serving as function approximator, enables powerful generalization. One of the key advantages of DRL is that it enables RL to scale to problems with high-dimensional state and action spaces. However, most existing successful DRL applications so far have been on visual domains (e.g., Atari games), and there is still a lot of work to be done for more realistic applications [25,26] with complex dynamics, which are not necessarily vision-based.DRL h...

show abstract

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

A survey and critique of multiagent deep reinforcement learning

Hernández-Leal¹,

Kartal²,

Taylor³

2019

Auton Agent Multi-Agent Syst

439

240

View full text Add to dashboard Cite

show abstract

“…And if not, can we estimate their utility functions solely from the agent's behaviour in a multi-objective decision problem? Albrecht and Stone recently published a comprehensive survey on opponent modelling for single-objective MAS [3]; many of the methods they surveyed could plausibly be adapted or extended to model other agents' intentions and utilities in MOMAS.…”

Section: Opponent Modelling and Modelling Opponent Utilitymentioning

confidence: 99%

Multi-objective multi-agent decision making: a utility-based analysis and survey

Rădulescu

Mannion

Roijers

et al. 2019

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

The majority of multi-agent system (MAS) implementations aim to optimise agents' policies with respect to a single objective, despite the fact that many real-world problem domains are inherently multi-objective in nature. Multiobjective multi-agent systems (MOMAS) explicitly consider the possible trade-offs between conflicting objective functions. We argue that, in MOMAS, such compromises should be analysed on the basis of the utility that these compromises have for the users of a system. As is standard in multi-objective optimisation, we model the user utility using utility functions that map value or return vectors to scalar values. This approach naturally leads to two different optimisation criteria: expected scalarised returns (ESR) and scalarised expected returns (SER). We develop a new taxonomy which classifies multi-objective multi-agent decision making settings, on the basis of the reward structures, and which and how utility functions are applied. This allows us to offer a structured view of the field, to clearly delineate the current state-of-the-art in multi-objective multi-agent decision making approaches and to identify promising directions for future research. Starting from the execu-

show abstract

“…Such stochastic elements can notably increase the complexity in multi-agent systems and multi-agent tasks, where agents learn to cooperate and compete simultaneously [6] [10]. As other agents adapt and actively adjust their policies, the best policy for each agent would evolve dynamically, giving rise to non-stationarity [8] [9]. In these studies, the cost of a trial to receive either a reward or punishment can be seen to be significant, and ideally, one would like to arrive at the correct conclusion by incurring minimum cost.…”

Section: Introductionmentioning

confidence: 99%

Leveraging Reinforcement Learning Techniques for Effective Policy Adoption and Validation

Kuang

Leung

2019

Computational Science and Its Applications – ICCSA 2019

View full text Add to dashboard Cite

Rewards and punishments in different forms are pervasive and present in a wide variety of decision-making scenarios. By observing the outcome of a sufficient number of repeated trials, one would gradually learn the value and usefulness of a particular policy or strategy. However, in a given environment, the outcomes resulting from different trials are subject to chance influence and variations. In learning about the usefulness of a given policy, significant costs are involved in systematically undertaking the sequential trials; therefore, in most learning episodes, one would wish to keep the cost within bounds by adopting learning stopping rules. In this paper, we examine the deployment of different stopping strategies in given learning environments which vary from highly stringent for mission critical operations to highly tolerant for non-mission critical operations, and emphasis is placed on the former with particular application to aviation safety. In policy evaluation, two sequential phases of learning are identified, and we describe the outcomes variations using a probabilistic model, with closedform expressions obtained for the key measures of performance. Decision rules that map the trial observations to policy choices are also formulated. In addition, simulation experiments are performed, which corroborate the validity of the theoretical results.

show abstract

Autonomous agents modelling other agents: A comprehensive survey and open problems

Cited by 300 publications

References 163 publications

A survey and critique of multiagent deep reinforcement learning

A survey and critique of multiagent deep reinforcement learning

Multi-objective multi-agent decision making: a utility-based analysis and survey

Leveraging Reinforcement Learning Techniques for Effective Policy Adoption and Validation

Contact Info

Product

Resources

About