Efficient Reinforcement Learning in Parameterized Models: Discrete Parameter Case

Regret Bounds for Reinforcement Learning with Policy Advice

“…1). Intuitively, the algorithm seeks to identify and use the policy in the input set Π that yields the highest average reward on the 5 One can easily prove that the upper bound H always exists for any unichain Markov reward process (see [12,Chap. 8]).…”

“…Nonetheless, the algorithm inherits the regret of UCRL itself and still displays a O(S √ A) dependency on states and actions. In [5] the Parameter Elimination (PEL) algorithm is provided with a set of MDPs. The algorithm is analyzed in the PAC-MDP framework and under the assumption that the true model actually belongs to the set of MDPs, it is shown to have a performance which does not depend on the size of the stateaction space and it only has a O( √ m) a dependency on the number of MDPs m. 8 In our setting, although no model is provided and no assumption on the optimality of π * is made, RLPA achieves the same dependency on m. The span sp(λ π ) of a policy is known to be a critical parameter determining how well and fast the average reward of a policy can be estimated using samples (see e.g., [1]).…”

“…In this case, an objective of maximizing the expected sum of rewards can be framed as planning in a finite-state partially observable Markov decision process [10]: if the parameter set is not too large, off-the-shelf POMDP planners can be used to yield significant performance improvements over state-of-the-art RL approaches [2]. Other work [5] on this setting has proved that the sample complexity of learning to act well scales independently of the size of the state and action space, and linearly with the size of the parameter set. These approaches focus on leveraging information about the model space in the context of Bayesian RL or PAC-style RL, in contrast to our model-free approach that focuses on regret.…”

Bayesian Reinforcement Learning with Exploration

“…We study the question of finding the minimax sample-complexity of reinforcement learning without making the usual Markov assumption, but where the learner has access to a finite set of reinforcement learning environments to which the truth is known to belong. This problem was tackled previously by Dyagilev et al (2008) and Lattimore et al (2013a). The new algorithm improves on the theoretical results in both papers and is simultaneously simpler and more elegant.…”

Action Time Sharing Policies for Ergodic Control of Markov Chains

“…[6]. For results on adaptive control in the non-countable setting we refer the reader to [7][8][9] and references therein: these deal with the classical setup, namely they seek a combined estimation and control scheme and consider parameterized models.…”