Bi-level Actor-Critic for Multi-agent Coordination

Zhang, Haifeng; Chen, Weizhe; Huang, Zeren; Li, Minne; Yang, Yaodong; Zhang, Weinan; Wang, Jun

doi:10.48550/arxiv.1909.03510

Cited by 4 publications

(6 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Actorcritic (AC) methods are a long-established class of techniques, consisting of two subproblems that are intertwined with each other. At the same time, AC methods are widely studied in recent years [147], [148], [146], [21], [69], [65], [23]. In the case of guaranteed convergence, [21] proposes a bi-level AC method to solve multi-agent reinforcement learning problem in finding Stackelberg equilibrium under Markov games.…”

Section: Deep Reinforcement Learningmentioning

confidence: 99%

“…At the same time, AC methods are widely studied in recent years [147], [148], [146], [21], [69], [65], [23]. In the case of guaranteed convergence, [21] proposes a bi-level AC method to solve multi-agent reinforcement learning problem in finding Stackelberg equilibrium under Markov games. It allows agents to have different knowledge base (thus intelligent), while their actions still can be executed simultaneously and distributedly.…”

Section: Deep Reinforcement Learningmentioning

confidence: 99%

“…Subsequently, it has been investigated in view of many important applications in various fields of science and engineering, particularly in economics, management, chem-istry, optimal control, and resource allocation problems [4], [5], [6], [7]. Especially, in recent years, a great amount of modern applications in the fields of machine learning and computer vision (e.g., hyper-parameter optimization [8], [9], [10], [11], multi-task and meta-learning [12], [13], [14], neural architecture search [15], [16], [17], generative adversarial learning [18], [19], [20], deep reinforcement learning [21], [22], [23] and image processing and analysis [24], [25], [26], just name a few) have arisen that fit the BLO framework.…”

Section: Introductionmentioning

confidence: 99%

“…In recent years, a variety of modern learning paradigms, including but not limited to, hyper-parameter optimization [24], [55], [56], [57]), multi-task and meta-learning [58], [59], [60], [61], neural architecture search [15], [17], [62], [63], adversarial learning [20], [19], [18], [64], and deep reinforcement learning [21], [65], have been investigated in machine learning and computer vision fields to address complex tasks in challenging real-world scenarios. For example, hyper-parameter optimization aims to jointly learn the model parameters and hyper-parameters [66].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond

Liu¹,

Gao²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Bi-Level Optimization (BLO) is originated from the area of economic game theory and then introduced into the optimization community. BLO is able to handle problems with a hierarchical structure, involving two levels of optimization tasks, where one task is nested inside the other. In machine learning and computer vision fields, despite the different motivations and mechanisms, a lot of complex problems, such as hyper-parameter optimization, multi-task and meta-learning, neural architecture search, adversarial learning and deep reinforcement learning, actually all contain a series of closely related subproblms. In this paper, we first uniformly express these complex learning and vision problems from the perspective of BLO. Then we construct a value-function-based single-level reformulation and establish a unified algorithmic framework to understand and formulate mainstream gradient-based BLO methodologies, covering aspects ranging from fundamental automatic differentiation schemes to various accelerations, simplifications, extensions and their convergence and complexity properties. Last but not least, we discuss the potentials of our unified BLO framework for designing new algorithms and point out some promising directions for future research.

show abstract

Section: Deep Reinforcement Learningmentioning

confidence: 99%

Section: Deep Reinforcement Learningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond

Liu¹,

Gao²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The bilevel constrained MDP is a special form of bilevel reinforcement learning [19], since 1) both their objectives are the summation of the discounted rewards in sequential states, 2) the analytic forms of objectives are unknown and they can only be learned through interactions with the environment in a model-free way. Their difference lies within the constraints which includes both states and policy in our model.…”

Section: Mathematical Modelmentioning

confidence: 99%

Bilevel Learning Model Towards Industrial Scheduling

Li¹,

Zhen²,

Yuan³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Automatic industrial scheduling, aiming at optimizing the sequence of jobs over limited resources, is widely needed in manufacturing industries. However, existing scheduling systems heavily rely on heuristic algorithms, which either generate ineffective solutions or compute inefficiently when job scale increases. Thus, it is of great importance to develop new large-scale algorithms that are not only efficient and effective, but also capable of satisfying complex constraints in practice. In this paper, we propose a Bilevel Deep reinforcement learning Scheduler, BDS, in which the higher level is responsible for exploring an initial global sequence, whereas the lower level is aiming at exploitation for partial sequence refinements, and the two levels are connected by a slidingwindow sampling mechanism. In the implementation, a Double Deep Q Network (DDQN) is used in the upper level and Graph Pointer Network (GPN) lies within the lower level. After the theoretical guarantee for the convergence of BDS, we evaluate it in an industrial automatic warehouse scenario, with job number up to 5000 in each production line. It is shown that our proposed BDS significantly outperforms two most used heuristics, three strong deep networks, and another bilevel baseline approach. In particular, compared with the most used greedy-based heuristic algorithm in real world which takes nearly an hour, our BDS can decrease the makespan by 27.5%, 28.6% and 22.1% for 3 largest datasets respectively, with computational time less than 200 seconds.

show abstract

$α^α$-Rank: Practically Scaling $α$-Rank through Stochastic Optimisation

Yang¹,

Tutunov²,

Sakulwongtana³

et al. 2019

Preprint

View full text Add to dashboard Cite

Recently, α-Rank 1 , a graph-based algorithm, has been proposed as a solution to ranking joint policy profiles in large scale multi-agent systems. α-Rank claimed tractability through a polynomial time implementation with respect to the total number of pure strategy profiles. Here, we note that inputs to the algorithm were not clearly specified in the original presentation. As such, we deem complexity claims as not grounded without a formal specification of the inputs. Using standard definitions from complexity theory, we demonstrate exponential complexity in terms of the number of agents and conjecture solving α-Rank is, in fact, NP-hard. After thorough discussions with the authors, we were suggested that the input to α-Rank can be an exponentially-sized payoff matrix; a claim promised to be clarified in subsequent manuscripts. Even though α-Rank exhibits a polynomial-time solution with respect to such an input, we, further, reflect additional critical problems. We demonstrate that due to the need of constructing an exponentially large Markov chain, α-Rank is infeasible beyond a small finite number of agents. We ground these claims by adopting amount of dollars spent as a non-refutable evaluation metric. Precisely, we demonstrate that with a one trillion dollars budget, α-Rank can only handle tens of agents. Realising such scalability issues, we present a stochastic implementation of α-Rank with a double oracle mechanism allowing for reductions in joint strategy spaces. Our method, named by α α -Rank, does not need to save exponentially-large transition matrices, and can terminate early by specifying a precision parameter. Although theoretically our method exhibits similar worst-case complexity guarantees compared to the default implementation of α-Rank, it allows us, for the first time, to practically conduct large-scale multi-agent evaluation experiments. Our results demonstrate that on 10 4 × 10 4 randomly generated matrices, we achieve 1000x speed reduction compared to α-Rank. Furthermore, we also show successful results on large joint strategy profiles with a maximum size in the order of O(2 25 ) (≈ 33 million joint strategies) -a setting not evaluable using α-Rank with reasonable computational budget.

show abstract

Bi-level Actor-Critic for Multi-agent Coordination

Cited by 4 publications

References 6 publications

Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond

Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond

Bilevel Learning Model Towards Industrial Scheduling

$α^α$-Rank: Practically Scaling $α$-Rank through Stochastic Optimisation

Contact Info

Product

Resources

About