Belief and truth in hypothesised behaviours

Abstract

There is a long history in game theory on the topic of Bayesian or "rational" learning, in which each player maintains beliefs over a set of alternative behaviours, or types, for the other players. This idea has gained increasing interest in the artificial intelligence (AI) community, where it is used as a method to control a single agent in a system composed of multiple agents with unknown behaviours. The idea is to hypothesise a set of types, each specifying a possible behaviour for the other agents, and to plan our own actions with respect to those types which we believe are most likely, given the observed actions of the agents. The game theory literature studies this idea primarily in the context of equilibrium attainment. In contrast, many AI applications have a focus on task completion and payoff maximisation. With this perspective in mind, we identify and address a spectrum of questions pertaining to belief and truth in hypothesised types. We formulate three basic ways to incorporate evidence into posterior beliefs and show when the resulting beliefs are correct, and when they may fail to be correct. Moreover, we demonstrate that prior beliefs can have a significant impact on our ability to maximise payoffs in the long-term, and that they can be computed automatically with consistent performance effects. Furthermore, we analyse the conditions under which we are able complete our task optimally, despite inaccuracies in the hypothesised types. Finally, we show how the correctness of hypothesised types can be ascertained during the interaction via an automated statistical analysis.

Learning Best Response Strategies for Agents in Ad Exchanges

“…The learning problem for this agent is made difficult by the fact that information is censored , i.e., the publisher knows if an impression is sold but no other quantitative information. We address this problem using the Harsanyi-Bellman Ad Hoc Coordination (HBA) algorithm [1,3], which conceptualises this interaction in terms of a Stochastic Bayesian Game and arrives at optimal actions by best responding with respect to probabilistic beliefs maintained over a candidate set of opponent behaviour profiles. We adapt and apply HBA to the censored information setting of ad exchanges.…”

“…We propose that this problem may be addressed by drawing on recent developments in machine learning, which allow tractable learning despite the incompleteness of models. In particular, we use the Harsanyi-Bellman Ad Hoc Coordination (HBA) algorithm [1,3], which conceptualises the interaction in terms of a space of 'types' (or opponent policies), over which the procedure maintains beliefs and uses the beliefs to guide optimal action selection. The attraction of this algorithm is that it can be shown to be optimal even when the hypothesised type space is not exactly correct but only approximates the possible behaviours of the opponent.…”

Can bounded and self-interested agents be teammates? Application to planning in ad hoc teams

“…This includes an algorithm for online planning in ad hoc teams (OPAT) [74] that solves a series of stage games assuming that other agents are optimal with the utility at each stage computed using Monte Carlo tree search. Albrecht and Ramamoorthy [6,8] generalize Bayesian games [43] to model the uncertainty about other agents' user-defined types and construct a Harsanyi-Bayesian ad hoc coordination game (HBA) that is solved online using learning. However, establishing common knowledge of the prior distribution over types to facilitate the solution of HBAs is problematic in ad hoc settings.…”

Towards a Fast Detection of Opponents in Repeated Stochastic Games

“…Recent work on Stochastic Bayesian Games has compared several ways to incorporate observations into beliefs over opponent types when those types are re-drawn after every state transition [1]. In contrast, we assume that opponents are redrawn for interactions over several repeated stochastic games.…”

“…Second, in an online phase a random process pairs the learning agent against the opponent for a stochastic game. 1 The learning agent has no control over this process and does not observe the opponent identity. When the game finishes the learning agent receives an observation (reward) and updates the belief accordingly.…”