Reinforcement Learning in Reward-Mixing MDPs

Abstract

Learning a near optimal policy in a partially observable system remains an elusive challenge in contemporary reinforcement learning. In this work, we consider episodic reinforcement learning in a reward-mixing Markov decision process (MDP). There, a reward function is drawn from one of multiple possible reward models at the beginning of every episode, but the identity of the chosen reward model is not revealed to the agent. Hence, the latent state space, for which the dynamics are Markovian, is not given to the agent. We study the problem of learning a near optimal policy for two reward-mixing MDPs. Unlike existing approaches that rely on strong assumptions on the dynamics, we make no assumptions and study the problem in full generality. Indeed, with no further assumptions, even for two switching reward-models, the problem requires several new ideas beyond existing algorithmic and analysis techniques for efficient exploration. We provide the first polynomial-time algorithm that finds an ǫ-optimal policy after exploring Õ(poly(H, ǫ −1 ) • S 2 A 2 ) episodes, where H is time-horizon and S, A are the number of states and actions respectively. This is the first efficient algorithm that does not require any assumptions in partially observed environments where the observation space is smaller than the latent state space.Algorithm 1 Learning Two Reward-Mixture MDPs 1: Run pure-exploration (Algorithm 2) to estimate second-order correlations 2: Estimate 2RM-MDP parameters M from the collected data (Algorithm 3) 3: Return π, the (approximately) optimal policy of M P m the probability of any event measured in the m th context (or in m th MDP). If probability of an event depends on a policy π, we add superscript π to P. We denote V π M as an expected long-term reward for model M with policy π. We use • to denote empirical counterparts. For any set A and d ∈ N, A d is a d-ary Cartesian product over A. We use a ∨ b to refer max(a, b) and a ∧ b to mean min(a, b) for a, b ∈ R. Specifically for M = 2: We use shorthand p m (x) := R m (r = 1|x) for m = 1, 2. Let an expected averaged reward p + (x) := 1 2 (p 1 (x) + p 2 (x)), differences in rewards p − (x) := 1 2 (p 1 (x) − p 2 (x)) and ∆(x) := |p − (x)|. AlgorithmsBefore developing a learning algorithm for 2RM-MDP, let us provide intuition behind our algorithm. At a high-level, our approach lies on the following observation:The latent reward model of 2RM-MDP can be recovered from reward correlations and the averaged reward.

Planning in Observable POMDPs in Quasipolynomial Time

“…Finally, [KECM21a] provides a computationally efficient learning algorithm for a special case of POMDPs which they call reward-mixing MDPs. Specifically, the POMDP is a disjoint union of two MDPs which are identical except for the rewards.…”

“…In particular, the convention in the recent theoretical RL literature on learning POMDPs [KAL16, GDB16, ALA16, JKKL20, XCGZ21, KECM21a, KECM21b] is to assume access to an oracle that solves the POMDP planning problem. There are some exceptions, but they require very strong assumptions on the model, which essentially trivialize the planning aspect -for instance, they assume that the state transitions are deterministic [KAL16,JKKL20], in which case (for known model) we always know the hidden state even without receiving any observations, or they assume that the "unobserved" portion of the state has constant size and never changes [KECM21a].…”

Learning in Observable POMDPs, without Computationally Intractable Oracles

“…[JKKL20, KAL16] proved polynomialtime learning results assuming deterministic transitions of the POMDP. [KECM21a] obtains results for learning latent MDPs which are a mixture of 2 underlying models; since the uncertainty in the system can be modeled by a single-dimensional parameter, their results are computationally efficient.…”

“…Planning algorithms in RL. The problem of planning in POMDPs (namely, finding the optimal policy when the model is known) has been extensively studied, with various proposed heuristics (e.g., [Mon82, CLZ97, HF00a, Hau00, RG02, TK03, PB04, SV05, PGT06, RPPCD08, SV10, SS12, SYHL13, GHL19, Han98, MKKC99, KMN99, LYX11, AYA18]), and a few provably efficient algorithms [BDRS96,KECM21a,GMR22]. Most closely related to our work is [GMR22], which shows a quasipolynomial-time planning algorithm for observable POMDPs.…”

“…Nevertheless there is a sizeable literature devoted to overcoming the statistical intractability of the learning problem by restricting to natural subclasses of POMDPs [KAL16, GDB16, ALA16, JKKL20, XCGZ21, KECM21a, KECM21b, LCSJ22]. There are far fewer works attempting to overcome computational intractability, and all make severe restrictions on either the model dynamics [JKKL20,KAL16] or the structure of the uncertainty [BDRS96,KECM21a]. The standard practice is to simply sidestep computational issues by assuming access to strong oracles such as ones that solve Optimistic Planning (given a constrained, non-convex set of POMDPs, find the maximum value achievable by any policy on any POMDP in the set) [JKKL20] or Optimistic Maximum Likelihood Estimation (given a set of action/observation sample trajectories, find a POMDP which obtains maximum value conditioned on approximately maximizing the likelihood of seeing those trajectories) [LCSJ22].…”