Open Problems in Cooperative AI

Abstract

Problems of cooperation-in which agents seek ways to jointly improve their welfare-are ubiquitous and important. They can be found at scales ranging from our daily routines-such as driving on highways, scheduling meetings, and working collaboratively-to our global challenges-such as peace, commerce, and pandemic preparedness. Arguably, the success of the human species is rooted in our ability to cooperate. Since machines powered by artificial intelligence are playing an ever greater role in our lives, it will be important to equip them with the capabilities necessary to cooperate and to foster cooperation.We see an opportunity for the field of artificial intelligence to explicitly focus effort on this class of problems, which we term Cooperative AI. The objective of this research would be to study the many aspects of the problems of cooperation and to innovate in AI to contribute to solving these problems. Central goals include building machine agents with the capabilities needed for cooperation, building tools to foster cooperation in populations of (machine and/or human) agents, and otherwise conducting AI research for insight relevant to problems of cooperation. This research integrates ongoing work on multi-agent systems, game theory and social choice, human-machine interaction and alignment, natural-language processing, and the construction of social tools and platforms. However, Cooperative AI is not the union of these existing areas, but rather an independent bet about the productivity of specific kinds of conversations that involve these and other areas. We see opportunity to more explicitly focus on the problem of cooperation, to construct unified theory and vocabulary, and to build bridges with adjacent communities working on cooperation, including in the natural, social, and behavioural sciences.Conversations on Cooperative AI can be organized in part in terms of the dimensions of cooperative opportunities. These include the strategic context, the extent of common versus conflicting interest, the kinds of entities who are cooperating, and whether researchers take the perspective of an individual or of a social planner. Conversations can also be focused on key capabilities necessary for cooperation, such as understanding, communication, cooperative commitments, and cooperative institutions. Finally, research should study the potential downsides of cooperative capabilities-such as exclusion and coercion-and how to channel cooperative capabilities to best improve human welfare. This research would connect AI research to the broader scientific enterprise studying the problem of cooperation, and to the broader social effort to solve cooperation problems. This conversation will continue at: www.cooperativeAI.com

Warmth and competence in human-agent cooperation

“…Competitive games have long been a focal point for AI research [12,74,75,83]. We follow recent calls to move AI research beyond competition and toward cooperation [16]. Most interaction research on deep reinforcement learning focuses on pure common-interest games such as Overcooked [13,81] and Hanabi [77], where coordination remains the predominant challenge.…”

“…However, many members of the public harbor doubts and concerns about the trustworthiness of AI [14,20,24,41]. This presents a pressing issue for cooperation between humans and AI agents [16].…”

How Can Creativity Occur in Multi-Agent Systems?

“…The are multi-agent RL methods that are intended to make learning cooperation between agents more efficient. The hope is that these methods will allow for better coordination and social welfare than training multiple agents using single-agent RL methods [4]. For example, Agence requires the group to work together to coordinate control of their planet to avoid falling to their doom.…”

Hidden Agenda: a Social Deduction Game with Diverse Learned Equilibria

“…Interacting and partnering with other agents offers substantial benefits [27]. It also carries considerable risk: though agents often share goals, they can also hold conflicting motives [7,8]. Multiagent research often studies the challenges of conflicting goals by assuming perfect information, but in real life interactants can conceal their goals and motivations [32].…”