Learning and Exploiting Relative Weaknesses of Opponent Agents

Abstract

Agents in a competitive interaction can greatly benefit from adapting to a particular adversary, rather than using the same general strategy against all opponents. One method of such adaptation is Opponent Modeling, in which a model of an opponent is acquired and utilized as part of the agent's decision procedure in future interactions with this opponent. However, acquiring an accurate model of a complex opponent strategy may be computationally infeasible. In addition, if the learned model is not accurate, then using it to predict the opponent's actions may potentially harm the agent's strategy rather than improving it. We thus define the concept of opponent weakness, and present a method for learning a model of this simpler concept. We analyze examples of past behavior of an opponent in a particular domain, judging its actions using a trusted judge. We then infer a weakness model based on the opponent's actions relative to the domain state, and incorporate this model into our agent's decision procedure. We also make use of a similar self-weakness model, allowing the agent to prefer states in which the opponent is weak and our agent strong; where we have a relative advantage over the opponent. Experimental results spanning two different test domains demonstrate the agents' improved performance when making use of the weakness models.

The adversarial activity model for bounded rational agents

“…As stated earlier, using a naive Eval maximization strategy to a certain search depth will not always yield satisfactory results for several reasons: (1) the search horizon problem when searching for a fixed depth [52]; (2) the strong assumption of an optimally rational, unbounded resources adversary; (3) using an estimated evaluation function which will not give optimal results in all world states and can be exploited [42].…”

“…However, its strict assumptions rarely hold for more complex competitive situations, where it might be too hard to come up with a good heuristic function and the agent's limited resources and computational limitation prevents it from expanding the full search tree (causing the well known horizon problem [52]). Moreover, running the algorithm under the assumptions that the opponent is rational and is using the same heuristic function might limit our outcome (see [42] for situations were those weaknesses were exploited); obviously, the more knowledge we have on our opponent the better we can strategise against him in an adversarial interaction (e.g. [9,11,60,67]).…”

“…Our experiments show that the agents who use our model outperformed the regular search agents in both domains and in a variety of evaluation functions and search depths. In addition, for the Connect Four board game, we present an analysis of player's behavior using a large set of completed matches log files and use it together with the results presented in [42] to further discuss the advantages of using the model in simple domains (the Connect Four domain) in which the classical search methods do provide a descent solution.…”

Learning to bid in bridge

“…One way to overcome this problem is to model only certain aspects of the agents' strategy. Markovitch and Reger (2005) learn the opponent's weaknesses and bias the search towards states where the opponent's strategy is expected to be weak.…”

“…This learning mechanism can be viewed as a speedup learner where a rough approximation of the other agents' decision procedure is learned and expressed by induced decision nets. Some of the above works include algorithms for learning opponent agent models on the basis of their past behavior (Billings et al, 2003;Davidson et al, 2000;Sen and Arora, 1997;Bruce et al, 2002;Markovitch & Reger, 2005). The learning framework presented here focuses on adapting to co-agents.…”