Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts

Zhang, Weinan; Wang, Xihuai; Shen, Jian; Zhou, Ming

doi:10.24963/ijcai.2021/466

Cited by 15 publications

(4 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In (Kamra et al 2020), the interaction graph-based trajectory prediction methods are suggested. In (Zhang et al 2021), a decen-tralized MBRL method is proposed with consideration of multiple opponent models. Unfortunately, these works either consider zero-sum games or treat the players as atomic and learn separate models for the players which could be computationally expensive.…”

Section: Related Workmentioning

confidence: 99%

Transition-Informed Reinforcement Learning for Large-Scale Stackelberg Mean-Field Games

Li,

Yu,

Wang

et al. 2024

AAAI

View full text Add to dashboard Cite

Many real-world scenarios including fleet management and Ad auctions can be modeled as Stackelberg mean-field games (SMFGs) where a leader aims to incentivize a large number of homogeneous self-interested followers to maximize her utility. Existing works focus on cases with a small number of heterogeneous followers, e.g., 5-10, and suffer from scalability issue when the number of followers increases. There are three major challenges in solving large-scale SMFGs: i) classical methods based on solving differential equations fail as they require exact dynamics parameters, ii) learning by interacting with environment is data-inefficient, and iii) complex interaction between the leader and followers makes the learning performance unstable. We address these challenges through transition-informed reinforcement learning. Our main contributions are threefold: i) we first propose an RL framework, the Stackelberg mean-field update, to learn the leader's policy without priors of the environment, ii) to improve the data efficiency and accelerate the learning process, we then propose the Transition-Informed Reinforcement Learning (TIRL) by leveraging the instantiated empirical Fokker-Planck equation, and iii) we develop a regularized TIRL by employing various regularizers to alleviate the sensitivity of the learning performance to the initialization of the leader's policy. Extensive experiments on fleet management and food gathering demonstrate that our approach can scale up to 100,000 followers and significantly outperform existing baselines.

show abstract

Section: Related Workmentioning

confidence: 99%

Transition-Informed Reinforcement Learning for Large-Scale Stackelberg Mean-Field Games

Li,

Yu,

Wang

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…From the analysis of [Zhang et al, 2021b], the sample efficiency of MARL can be decomposed into two parts, i.e., dynamics sample complexity, which measures the amount of interactions with the real environment, and the opponent sample complexity, which measures the amount of interactions between the ago agent i and other agents {−i}. With this regard, it is natural to derive the value discrepancy of the agent policy in the multi-agent environment model (i.e., with the state dynamics model and the opponent models) and the real environment with respect to the error terms of the state dynamics model and opponent models when training the policy in via Dyna-style model rollout.…”

Section: Multi-agent Rlmentioning

confidence: 99%

“…With this regard, it is natural to derive the value discrepancy of the agent policy in the multi-agent environment model (i.e., with the state dynamics model and the opponent models) and the real environment with respect to the error terms of the state dynamics model and opponent models when training the policy in via Dyna-style model rollout. The bound shows that the opponent models with higher modeling error contribute larger to the discrepancy bound, which motivates the design of the algorithm called adaptive opponent-wise rollout policy optimization (AORPO) [Zhang et al, 2021b]. Specifically, the rollout scheme of AORPO allows the opponent models with lower generalization errors to sample longer trajectories while the shorter sampled trajectories can be supplemented with a communication protocol with real opponents.…”

Section: Multi-agent Rlmentioning

confidence: 99%

A Survey on Model-based Reinforcement Learning

Fan-Ming¹,

Xu²,

Lai³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Reinforcement learning (RL) solves sequential decision-making problems via a trial-and-error process interacting with the environment. While RL achieves outstanding success in playing complex video games that allow huge trial-and-error, making errors is always undesired in the real world. To improve the sample efficiency and thus reduce the errors, model-based reinforcement learning (MBRL) is believed to be a promising direction, which builds environment models in which the trial-and-errors can take place without real costs. In this survey, we take a review of MBRL with a focus on the recent progress in deep RL. For non-tabular environments, there is always a generalization error between the learned environment model and the real environment. As such, it is of great importance to analyze the discrepancy between policy training in the environment model and that in the real environment, which in turn guides the algorithm design for better model learning, model usage, and policy training. Besides, we also discuss the recent advances of model-based techniques in other forms of RL, including offline RL, goal-conditioned RL, multi-agent RL, and meta-RL. Moreover, we discuss the applicability and advantages of MBRL in real-world tasks. Finally, we end this survey by discussing the promising prospects for the future development of MBRL. We think that MBRL has great potential and advantages in real-world applications that were overlooked, and we hope this survey could attract more research on MBRL.

show abstract

“…In an adaptive system [29], an agent is constantly learning and making dynamic adaptive adjustments. An opponent model should be constructed to enable adaptive changes to be made based on the opponent's real-time strategy.…”

Section: Introductionmentioning

confidence: 99%

Improving Agent Decision Payoffs via a New Framework of Opponent Modeling

et al. 2023

View full text Add to dashboard Cite

The payoff of an agent depends on both the environment and the actions of other agents. Thus, the ability to model and predict the strategies and behaviors of other agents in an interactive decision-making scenario is one of the core functionalities in intelligent systems. State-of-the-art methods for opponent modeling mainly use an explicit model of opponents’ actions, preferences, targets, etc., that the primary agent uses to make decisions. It is more important for an agent to increase its payoff than to accurately predict opponents’ behavior. Therefore, we propose a framework synchronizing the opponent modeling and decision making of the primary agent by incorporating opponent modeling into reinforcement learning. For interactive decisions, the payoff depends not only on the behavioral characteristics of the opponent but also the current state. However, confounding the two obscures the effects of state and action, which then cannot be accurately encoded. To this end, state evaluation is separated from action evaluation in our model. The experimental results from two game environments, a simulated soccer game and a real game called quiz bowl, show that the introduction of opponent modeling can effectively improve decision payoffs. In addition, the proposed framework for opponent modeling outperforms benchmark models.

show abstract

Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts

Cited by 15 publications

References 3 publications

Transition-Informed Reinforcement Learning for Large-Scale Stackelberg Mean-Field Games

Transition-Informed Reinforcement Learning for Large-Scale Stackelberg Mean-Field Games

A Survey on Model-based Reinforcement Learning

Improving Agent Decision Payoffs via a New Framework of Opponent Modeling

Contact Info

Product

Resources

About