Online Markov Decision Processes

Abstract

We consider a Markov decision process (MDP) setting in which the reward function is allowed to change after each time step (possibly in an adversarial manner), yet the dynamics remain fixed. Similar to the experts setting, we address the question of how well an agent can do when compared to the reward achieved under the best stationary policy over time. We provide efficient algorithms, which have regret bounds with no dependence on the size of state space. Instead, these bounds depend only on a certain horizon time of the process and logarithmically on the number of actions.1. Introduction. Finite state and actions Markov decision processes (MDPs) are a popular and attractive way to formulate many stochastic optimization problems ranging from robotics to finance (Puterman [17], Bertsekas and Tsitsiklis [2], Sutton and Barto [18]). Unfortunately, in many applications the Markovian assumption made is only a relaxation of the real model. A popular framework that is not Markovian is the experts problem, in which during every round a learner chooses one of n decision-making experts and incurs the loss of the chosen expert. The setting is typically an adversarial one, where Nature provides the examples to a learner. The standard objective here is a myopic, backwards-looking one-in retrospect, we desire that our performance is not much worse than had we chosen any single expert on the sequence of examples provided by Nature. Expert algorithms have played an important role in computer science in the past decade, solving problems varying from classification to online portfolios (see Littlestone and Warmuth [13], Blum and Kalai [3], Helmbold et al. [8]).There is an inherent tension between the objectives in an expert setting and those in a reinforcement learning (RL) setting. In contrast to the myopic nature of the expert algorithms, an RL setting typically makes the much stronger assumption of a fixed environment, and the forward-looking objective is to maximize some measure of the future reward with respect to this fixed environment. Therefore, in RL the past actions have a major influence on the current reward, whereas in the regret setting they have no influence. In this paper, we relax the Markovian assumption of the MDPs by letting the reward function be time dependent, and even chosen by an adversary as is done in the expert setting, but still keeping the underlying structure of an MDP.The motivation of this work is to understand how to efficiently incorporate the benefits of existing experts' algorithms into a more adversarial reinforcement learning setting, where certain aspects of the environment could change over time. A naive way to implement an experts' algorithm is to simply associate an expert with each fixed policy. The running time of such algorithms is polynomial in the number of experts, and the regret (the difference from the optimal reward) is logarithmic in the number of experts. For our setting, the number of policies is huge, namely, for an MDP with state space S and action space A we have A S polic...

Chasing Ghosts: Competing with Stateful Policies

“…Our setting might seem reminiscent of (online) reinforcement learning models, and in particular, of online Markov Decision Processes (MDPs) (Even-Dar et al, 2009;Neu et al, 2010). In these models, there is typically a finite number of states, and the player's actions on each round cause him to transition from one state to another.…”

A hybrid controller based on the egocentric perceptual principle

“…EA-EMT is unlike other adaptive control algorithms based on expert ensembles, where experts directly produce actions or plans to be fused (e.g. [27,28,16]). 3 Rather, EA-EMT operates in two distinct modules: the expert-based model estimation and a control algorithm that utilises that model.…”

Algorithms for Reinforcement Learning

“…The main lesson here is that off-line planning in the worst-case can scale exponentially with the dimensionality of the state space (Chow and Tsitsiklis, 1989), while online planning (i.e., planning for the "current state") can break the curse of dimensionality by amortizing the planning effort over multiple time steps (Rust, 1996;Szepesvári, 2001). Other topics of interest include the linear programming-based approaches (de Farias and Van Roy, 2003, 2006, dual dynamic programming (Wang et al, 2008), techniques based on sample average approximation (Shapiro, 2003) such as PEGASUS (Ng and Jordan, 2000), online learning in MDPs with arbitrary reward processes (Even-Dar et al, 2005;Neu et al, 2010), or learning with (almost) no restrictions in a competitive framework (Hutter, 2004). Other important topics include learning and acting in partially observed MDPs (for recent developments, see, e.g., Littman et al, 2001;Toussaint et al, 2008;, learning and acting in games or under some other optimization criteria (Littman, 1994;Heger, 1994;Szepesvári and Littman, 1999;Borkar and Meyn, 2002), or the development of hierarchical and multi-time-scale methods (Dietterich, 1998;Sutton et al, 1999b).…”