Quantum Algorithms for Reinforcement Learning with a Generative Model

Abstract

Reinforcement learning studies how an agent should interact with an environment to maximize its cumulative reward. A standard way to study this question abstractly is to ask how many samples an agent needs from the environment to learn an optimal policy for a γ-discounted Markov decision process (MDP). For such an MDP, we design quantum algorithms that approximate an optimal policy (π * ), the optimal value function (v * ), and the optimal Q-function (q * ), assuming the algorithms can access samples from the environment in quantum superposition. This assumption is justified whenever there exists a simulator for the environment; for example, if the environment is a video game or some other program. Our quantum algorithms, inspired by value iteration, achieve quadratic speedups over the best-possible classical sample complexities in the approximation accuracy ( ) and two main parameters of the MDP: the effective time horizon ( 1 1−γ ) and the size of the action space (A). Moreover, we show that our quantum algorithm for computing q * is optimal by proving a matching quantum lower bound.

Quantum Multi-Armed Bandits and Stochastic Linear Bandits Enjoy Logarithmic Regrets

“…For bandit problems, Casalé et al (Casalé et al 2020) initiated the study of quantum algorithms for best-arm identification of MAB, and Wang et al (Wang et al 2021b) proved optimal results for best-arm identification of MAB with Bernoulli arms. As an extension, Wang et al (Wang et al 2021a) proposed quantum algorithms for finding an optimal policy for a Markov decision process with quantum speedup. These results focused on the exploration of reinforcement learning models, and in terms of the tradeoff between exploration and exploitation, the only work we are aware of is (Lumbreras, Haapasalo, and Tomamichel 2022), which proved that the regret of online learning of properties of quantum states has lower bounds Ω( √ T ).…”

Quantum Multi-Armed Bandits and Stochastic Linear Bandits Enjoy Logarithmic Regrets

“…For bandit problems, Casalé et al [9] initiated the study of quantum algorithms for best-arm identification of MAB, and Wang et al [26] proved optimal results for best-arm identification of MAB with Bernoulli arms. As an extension, Wang et al [25] proposed quantum algorithms for finding an optimal policy for a Markov decision process with quantum speedup. These results focused on exploration of reinforcement learning models, and in terms of the tradeoff between exploration and exploitation, the only work we are aware of is [18], which proved that the regret of online learning of properties of quantum states has lower bounds Ω( √ T ).…”