Optimal Payoff Functions for Members of Collectives

Abstract

We consider the problem of designing (perhaps massively distributed) collectives of computational processes to maximize a provided "world" utility function. We consider this problem when the behavior of each process in the collective can be cast as striving to maximize its own payoff utility function. For such cases the central design issue is how to initialize/update those payoff utility functions of the individual processes so as to induce behavior of the entire collective having good values of the world utility. Traditional "team game" approaches to this problem simply assign to each process the world utility as its payoff utility function. In previous work we used the "Collective Intelligence" (COIN) framework to derive a better choice of payoff utility functions, one that results in world utility performance up to orders of magnitude superior to that ensuing from use of the team game utility. In this paper we extend these results using a novel mathematical framework. We review the derivation under that new framework of the general class of payoff utility functions that both i) are easy for the individual processes to try to maximize, and ii) have the property that if good values of them are achieved, then we are assured of a high value of world utility. These are the "Aristocrat Utility" and a new variant of the "Wonderful Life Utility" that was introduced in the previous COIN work. We demonstrate experimentally that using these new utility functions can result in significantly improved performance over that of previously investigated COIN payoff utilities, over and above those previous utilities' superiority to the conventional team game utility. These results also illustrate the substantial superiority of these payoff functions to the perhaps the most natural version of the economics technique of "endogenizing externalities".

Handling Communication Restrictions and Team Formation in Congestion Games

“…In this section, we first summarize the formalization of the concepts of factoredness and learnability which are essential in deriving good private utilities for the agents [37]. We then present a class of private utilities satisfying those two properties and discuss four variants (based on different trade-offs of learnability vs. degree of factoredness) applicable to domains with communication restrictions (Section 2.3).…”

“…All the components of z that are affected by agent i are replaced with the fixed constant c i . Such difference utilities are fully factored no matter what the choice of c i , because the second term does not depend on agent i's actions [34,37]. Furthermore, they usually have better learnability than does a team game, because of the second term of D, which removes a lot of the effect of other agents (i.e., noise) from agent i's evaluation function.…”

“…Many methods exist for coordinating the actions of autonomous agents in a large multi-agent system when those agents can fully communicate with one another [7,11,21,27,33,37]. One particular solution to that problem is given within the framework of "collectives" defined as large multi-agent systems where there is a well-defined "system utility" function which rates the possible dynamic histories of the collection, and where each agent is only concerned with maximizing its own "private utility" function [37].…”

“…One particular solution to that problem is given within the framework of "collectives" defined as large multi-agent systems where there is a well-defined "system utility" function which rates the possible dynamic histories of the collection, and where each agent is only concerned with maximizing its own "private utility" function [37]. However, many problems impose communication restrictions among the agents, rendering the coordination problem more difficult [6].…”

COllective INtelligence with Sequences of Actions

“…In this Section, we briefly outline the theory of COIN as developed by Wolpert et al More elaborate details can be found in [21,17,18]. Broadly speaking, COIN defines the conditions that an agent's private utility function has to meet to increase the probability that learning to optimize this function leads to increased performance of the collective of agents.…”