No-regret learning in convex games

Abstract

Quite a bit is known about minimizing different kinds of regret in experts problems, and how these regret types relate to types of equilibria in the multiagent setting of repeated matrix games. Much less is known about the possible kinds of regret in online convex programming problems (OCPs), or about equilibria in the analogous multiagent setting of repeated convex games. This gap is unfortunate, since convex games are much more expressive than matrix games, and since many important machine learning problems can be expressed as OCPs. In this paper, we work to close this gap: we analyze a spectrum of regret types which lie between external and swap regret, along with their corresponding equilibria, which lie between coarse correlated and correlated equilibrium. We also analyze algorithms for minimizing these regret types. As examples of our framework, we derive algorithms for learning correlated equilibria in polyhedral convex games and extensive-form correlated equilibria in extensive-form games. The former is exponentially more efficient than previous algorithms, and the latter is the first of its type.

Towards Minimizing Disappointment in Repeated Games

“…While there are many algorithms that are guaranteed to achieve no regret (e.g., Bowling, 2004;Foster & Vohra, 1999;Gordon et al, 2008), we show that it is impossible for an algorithm to be guaranteed to have no disappointment against an unknown associate. However, it is possible for an algorithm to quickly achieve and maintain low disappointment against classes of algorithms in many repeated games.…”

“…Agent i is said to have no regret when lim T →∞R T i ≤ 0. When all agents use no-regret learning rules, play converges to correlated equilibria (Greenwald & Jafari, 2003;Gordon et al, 2008).…”

“…Several metrics have been defined and used in the literature to measure the success of an expert algorithm in this process, perhaps the most popular of which is regret (Foster & Vohra, 1999;Greenwald & Jafari, 2003;Bowling, 2004;Gordon, Greenwald, & Marks, 2008). Loosely, an expert algorithm's (external) regret is the difference between the payoffs the agent would have received had it always followed its best expert against its associates' observed actions and the payoffs it actually received.…”

Computer science and decision theory

“…Other equilibrium concepts that provide computational challenges include correlated equilibria, cooperative equilibria, and evolutionary stable strategies [14,145,423]. Other topics of interest are learning in repeated games and regret minimizing algorithms [146,153,174] and games with state and their connections to reinforcement learning [50,61,229]. Finally, cryptographic methods are needed to make sure that in the course of a game, a player's views, utilities, preferences, strategies are kept private.…”

Decision Making in Uncertain and Changing Environments

“…In fact, this is an impossible task if one does not add priors, which is equivalent to adding structure on the environment. Since Nature's complexity is unbounded, even a very patient 1 Littlestone and Warmuth (1994); Cesa-Bianchi et al (1996); Vovk (1998); Auer and Long (1999); Foster and Vohra (1999); Freund and Schapire (1999); Lugosi (2003, 2006); Greenwald and Jafari (2003); Cesa-Bianchi et al (2007); Gordon et al (2008). 2 Hannan (1957); Foster and Vohra (1993, 1997, 1998; Levine (1995, 1999);Hart andMas-Colell (2000, 2001a); Lehrer (2003); Hart (2005).…”