A Finite Memory Automaton for Two-Armed Bernoulli Bandit Problems

Abstract

Existing approaches to the multi-armed bandit (MAB) primarily rely on perfect recall of past actions to generate estimates for arm payoff probabilities; it is further assumed that the decision maker knows whether arm payoff probabilities can change. To capture the computational limitations many decision making systems face, we explore performance under bounded resources in the form of imperfect recall of past information. We present a finite memory automaton (FMA) designed to solve static and dynamic MAB problems. The FMA demonstrates that an agent can learn a low regret strategy without knowing whether arm payoff probabilities are static or dynamic and without having perfect recall of past actions. Roughly speaking, the automaton works by maintaining a relative ranking of arms rather than estimating precise payoff probabilities.

On composition of bounded-recall plans

“…Finite state machines have been previously used in game theory to model players with bounded rationality in iterative strategic games [6,7,8,9,10], to model the evolution of rational players [11], to analyze two-armed Bernoulli bandit problems [12], to model AI agents behavior in video games [13], and to specify AI agents for border patrol [14]. Kanovich, Kirgin, Nigam, and Scedrov proved NPcompleteness of a security problem in collaborative systems with bounded-recall agents [15].…”