Online Learning for Unknown Partially Observable MDPs

Abstract

Solving Partially Observable Markov Decision Processes (POMDPs) is hard. Learning optimal controllers for POMDPs when the model is unknown is harder. Online learning of optimal controllers for unknown POMDPs, which requires efficient learning using regret-minimizing algorithms that effectively tradeoff exploration and exploitation, is even harder, and no solution exists currently. In this paper, we consider infinite-horizon average-cost POMDPs with unknown transition model, though known observation model. We propose a natural posterior sampling-based reinforcement learning algorithm (POMDP-PSRL) and show that it achieves O(T 2/3 ) regret where T is the time horizon. To the best of our knowledge, this is the first online RL algorithm for POMDPs and has sub-linear regret.

Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes

“…Reinforcement learning in POMDPs. Our work is related to the recent line of research on developing provably efficient online RL methods for POMDPs (Guo et al, 2016;Krishnamurthy et al, 2016;Jin et al, 2020;Xiong et al, 2021;Jafarnia-Jahromi et al, 2021;Efroni et al, 2022;Liu et al, 2022). In the online setting, the actions are specified by history-dependent policies and thus the latent state does not directly affect the actions.…”