No-Regret Learning and a Mechanism for Distributed Multiagent Planning

Abstract

We develop a novel mechanism for coordinated, distributed multiagent planning. We consider problems stated as a collection of single-agent planning problems coupled by common soft constraints on resource consumption. (Resources may be real or fictitious, the latter introduced as a tool for factoring the problem). A key idea is to recast the distributed planning problem as learning in a repeated game between the original agents and a newly introduced group of adversarial agents who influence prices for the resources. The adversarial agents benefit from arbitrage: that is, their incentive is to uncover violations of the resource usage constraints and, by selfishly adjusting prices, encourage the original agents to avoid plans that cause such violations. If all agents employ no-regret learning algorithms in the course of this repeated interaction, we are able to show that our mechanism can achieve design goals such as social optimality (efficiency), budget balance, and Nash-equilibrium convergence to within an error which approaches zero as the agents gain experience. In particular, the agents' average plans converge to a socially optimal solution for the original planning task. We present experiments in a simulated network routing domain demonstrating our method's ability to reliably generate sound plans.

Lagrangian Relaxation for Large-Scale Multi-agent Planning

“…As far as multi-agent planning is concerned, researchers have previously used Lagrangian relaxation with some success; however, past work often either assumes infinitely divisible resources [15], [4], or tries to get an exact solution and therefore doesn't scale to really large problems [16]. Researchers have also studied restrictions to the type of planning problem considered, in some cases providing strong approximation guarantees using polynomialtime algorithms [17], [1].…”

Lazy Auctions for Multi-robot Collision Avoidance and Motion Control under Uncertainty

“…This is in contrast to another large body of works on coordination that focusses on agent interaction via objective functions (e.g. [13,14,6]). Since our model is based on BLPs we can employ our method in the context of particle-based multi-robot open-loop control [5].…”

Online Optimisation for Online Learning and Control -- From No-Regret to Generalised Error Convergence

“…No-regret bounds and algorithms have been studied and deployed in a great many online learning scenarios, including, among others, time-series prediction in ARMA models [2], multi-agent coordination [9], game-theory [14,6,21]. For a classic text book, the reader is referred to [11], whereas a recent survey can be found in [18].…”

“…If a no-external regret bound is achieved then, provided the prediction loss is convex in the parameters, (sub-) optimality guarantees can be given to bound the average prediction errors and the degree of sub-optimality of the parametric predictor whose parameter is the average of all choices of the adapted online predictors' parameters. This has given rise to algorithms that are no longer purely online learning and that decompose learning and prediction into two phases: a learning phase where the online learning method is employed to adapt the learner's parameter online for some time and a subsequent prediction phase making use of the average parameter obtained from the learning phase [11,9].…”