Scalable Multi-Agent Reinforcement Learning for Warehouse Logistics with Robotic and Human Co-Workers

Krnjaic, Aleksandar; Thomas, J. Hywel; Papoudakis, Georgios; Schäfer, Lukas; Börsting, Peter; Albrecht, Stefano V.

doi:10.48550/arxiv.2212.11498

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, redundant computation, which implies that the same or similar computation is performed redundantly by different agents, is a well-known traditional problem in MASs [19] and comes from overlapping observations, leading to similar associated processes in several agents. This problem widely exists in applications such as distributed robot systems [20], distributed sensor networks [19,21] and distributed air traffic control [22], where there is a natural spatial distribution of information and where the agents have many overlapping observations. As shown in Figure 1a, the observations of the decentralized executing agents contain numerous overlapping regions.…”

Section: Introductionmentioning

confidence: 99%

Locally Centralized Execution for Less Redundant Computation in Multi-Agent Cooperation

Bai,

Sugawara

2024

Information

View full text Add to dashboard Cite

Decentralized execution is a widely used framework in multi-agent reinforcement learning. However, it has a well-known but neglected shortcoming, redundant computation, that is, the same/similar computation is performed redundantly in different agents owing to their overlapping observations. This study proposes a novel method, the locally centralized team transformer (LCTT), to address this problem. This method first proposes a locally centralized execution framework that autonomously determines some agents as leaders that generate instructions and other agents as workers to act according to the received instructions without running their policy networks. For the LCTT, we subsequently propose the team-transformer (T-Trans) structure, which enables leaders to generate targeted instructions for each worker, and the leadership shift, which enables agents to determine those that should instruct or be instructed by others. The experimental results demonstrated that the proposed method significantly reduces redundant computations without decreasing rewards and achieves faster learning convergence.

show abstract

Section: Introductionmentioning

confidence: 99%

Locally Centralized Execution for Less Redundant Computation in Multi-Agent Cooperation

Bai,

Sugawara

2024

Information

View full text Add to dashboard Cite

show abstract

“…Cooperative multi-agent reinforcement learning (MARL) jointly trains a team of agents to exhibit behaviour which maximises shared cumulative rewards. MARL can tackle problems such as autonomous driving (Shalev-Shwartz et al, 2016;Zhou et al, 2021) and warehouse logistics (Li et al, 2019;Krnjaic et al, 2022), but its real-world adaptation is still limited. Two remaining challenges of MARL are the large number of samples required to learn cooperation and the non-stationarity of the optimisation due to agents learning simultaneously (Papoudakis et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

Ensemble Value Functions for Efficient Exploration in Multi-Agent Reinforcement Learning

Schäfer¹,

Oliver²,

McAleer³

et al. 2023

Preprint

View full text Add to dashboard Cite

Cooperative multi-agent reinforcement learning (MARL) requires agents to explore to learn to cooperate. Existing value-based MARL algorithms commonly rely on random exploration, such as -greedy, which is inefficient in discovering multiagent cooperation. Additionally, the environment in MARL appears non-stationary to any individual agent due to the simultaneous training of other agents, leading to highly variant and thus unstable optimisation signals. In this work, we propose ensemble value functions for multi-agent exploration (EMAX), a general framework to extend any value-based MARL algorithm. EMAX trains ensembles of value functions for each agent to address the key challenges of exploration and non-stationarity: (1) The uncertainty of value estimates across the ensemble is used in a UCB policy to guide the exploration of agents to parts of the environment which require cooperation. (2) Average value estimates across the ensemble serve as target values. These targets exhibit lower variance compared to commonly applied target networks and we show that they lead to more stable gradients during the optimisation. We instantiate three value-based MARL algorithms with EMAX, independent DQN, VDN and QMIX, and evaluate them in 21 tasks across four environments. Using ensembles of five value functions, EMAX improves sample efficiency and final evaluation returns of these algorithms by 54%, 55%, and 844%, respectively, averaged all 21 tasks.

show abstract

Scalable Multi-Agent Reinforcement Learning for Warehouse Logistics with Robotic and Human Co-Workers

Cited by 2 publications

References 0 publications

Locally Centralized Execution for Less Redundant Computation in Multi-Agent Cooperation

Locally Centralized Execution for Less Redundant Computation in Multi-Agent Cooperation

Ensemble Value Functions for Efficient Exploration in Multi-Agent Reinforcement Learning

Contact Info

Product

Resources

About