Decision-Theoretic Planning Under Anonymity in Agent Populations

Abstract

We study the problem of self-interested planning under uncertainty in settings shared with more than a thousand other agents, each of which plans at its own individual level. We refer to such large numbers of agents as an agent population. The decision-theoretic formalism of interactive partially observable Markov decision process (I-POMDP) is used to model the agent's self-interested planning. The first contribution of this article is a method for drastically scaling the finitely-nested I-POMDP to certain agent populations for the first time. Our method exploits two types of structure that is often exhibited by agent populations -- anonymity and context-specific independence. We present a variant called the many-agent I-POMDP that models both these types of structure to plan efficiently under uncertainty in multiagent settings. In particular, the complexity of the belief update and solution in the many-agent I-POMDP is polynomial in the number of agents compared with the exponential growth that challenges the original framework. While exploiting structure helps mitigate the curse of many agents, the well-known curse of history that afflicts I-POMDPs continues to challenge scalability in terms of the planning horizon. The second contribution of this article is an application of the branch-and-bound scheme to reduce the exponential growth of the search tree for look ahead. For this, we introduce new fast-computing upper and lower bounds for the exact value function of the many-agent I-POMDP. This speeds up the look-ahead computations without trading off optimality, and reduces both memory and run time complexity. The third contribution is a comprehensive empirical evaluation of the methods on three new problems domains -- policing large protests, controlling traffic congestion at a busy intersection, and improving the AI for the popular Clash of Clans multiplayer game. We demonstrate the feasibility of exact self-interested planning in these large problems, and that our methods for speeding up the planning are effective. Altogether, these contributions represent a principled and significant advance toward moving self-interested planning under uncertainty to real-world applications.

Scalable Decision-Theoretic Planning in Open and Typed Multiagent Systems

“…We observe that in many open domains such as wildfire suppression, which particular firefighter is performing an action is not relevant to the decision making. Similar observations have been made in other domains (Velagapudi et al 2011;Nguyen, Kumar, and Lau 2017;Sonu, Chen, and Doshi 2017). Consequently, further efficiency is made possible by exploiting the property of anonymity of the other agents (Sonu, Chen, and Doshi 2017).…”

“…Similar observations have been made in other domains (Velagapudi et al 2011;Nguyen, Kumar, and Lau 2017;Sonu, Chen, and Doshi 2017). Consequently, further efficiency is made possible by exploiting the property of anonymity of the other agents (Sonu, Chen, and Doshi 2017). These modeling approaches are utilized in a new generalization of Monte Carlo tree search (Silver and Veness 2010) to plan in open, many-agent settings.…”

“…Therefore, any k agents of the same frame identically impact the fire. Such domains exhibit frame-action anonymity (Sonu, Chen, and Doshi 2017).…”

“…Formally, let configuration C = n a1,θ1 , ..., n a |A| ,θ |Θ| , where n a1,θ1 is the number of agents with frame θ 1 performing action a 1 in a joint action tuple (a i , a −i ). Histories with configurations (rather than joint actions) are now sufficient statistics (Sonu, Chen, and Doshi 2017) and the representation of T i , O i , and R i can be greatly compacted since the number of possible configurations |C| ∝ |Ag|+m+1 m+1 is polynomial in the number of agents and thus much fewer than the exponential number of possible joint actions m |Ag| , where m = max{|A i |, |A j |, . .…”

“…As scalability is a general and ever-present challenge for multiagent planning, investigations have first focused on multiagent planning for smaller groups of agents, e.g., (Amato and Oliehoek 2015;Claes et al 2017;Hoang and Low 2013;Nair et al 2005;Oliehoek et al 2008). Subsequently, there has been some focus on scalable planning in large groups of agents (Oliehoek, Whiteson, and Spaan 2013;Nguyen, Kumar, and Lau 2017;Sonu, Chen, and Doshi 2017;Velagapudi et al 2011) albeit, in the absence of openness. Indeed, state-of-the-art multiagent planning in open environments has demonstrated successful planning with up to 5 agents (Chandrasekaran et al 2016;Cohen, Dibangoye, and Mouaddib 2017).…”