We propose two algorithms for episodic stochastic shortest path problems with linear function approximation. The first is computationally expensive but provably obtains Õ( B 3 d 3 K/c min ) regret, where B is a (known) upper bound on the optimal cost-to-go function, d is the feature dimension, K is the number of episodes, and c min is the minimal cost of non-goal state-action pairs (assumed to be positive). The second is computationally efficient in practice, and we conjecture that it obtains the same regret bound. Both algorithms are based on an optimistic least-squares version of value iteration analogous to the finite-horizon backward induction approach from Jin et al. [2020]. To the best of our knowledge, these are the first regret bounds for stochastic shortest path that are independent of the size of the state and action spaces.

No abstract

For the misspecified linear Markov decision process (MLMDP) model of Jin et al. [2020], we propose an algorithm with three desirable properties. (P1) Its regret after K episodes scales as K max{ε mis , ε tol }, where ε mis is the degree of misspecification and ε tol is a user-specified error tolerance. (P2) Its space and per-episode time complexities remain bounded as K → ∞. (P3) It does not require ε mis as input. To our knowledge, this is the first algorithm satisfying all three properties. For concrete choices of ε tol , we also improve existing regret bounds (up to log factors) while achieving either (P2) or (P3) (existing algorithms satisfy neither). At a high level, our algorithm generalizes (to MLMDPs) and refines the Sup-Lin-UCB algorithm, which Takemura et al. [2021] recently showed satisfies (P3) in the contextual bandit setting.