Stochastic Shortest Path with Adversarially Changing Costs

Rosenberg, Aviv; Mansour, Yishay

doi:10.24963/ijcai.2021/404

Cited by 7 publications

(12 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In general, the costs may be chosen to be known (as in our case and Rosenberg et al [25]), unknown and requiring estimation as in Chen et al [7], or adversarial as in Neu et al [21], Rosenberg and Mansour [24], Chen et al [8]. The adversarial setting is quite distinct, while the known and unknown costs cases have similar approaches.…”

Section: Stochastic Shortest Paths With Unknown Transitionsmentioning

confidence: 99%

Convex duality for stochastic shortest path problems in known and unknown environments

Francis-Staite¹

2022

Preprint

View full text Add to dashboard Cite

This paper studies Stochastic Shortest Path (SSP) problems in known and unknown environments from the perspective of convex optimisation. It first recalls results in the known parameter case, and develops understanding through different proofs. It then focuses on the unknown parameter case, where it studies extended value iteration (EVI) operators. This includes the existing operators used in Rosenberg et al. [25] and Tarbouriech et al. [30] based on the 1 norm and supremum norm, as well as defining EVI operators corresponding to other norms and divergences, such as the KL-divergence. This paper shows in general how the EVI operators relate to convex programs, and the form of their dual, where strong duality is exhibited.This paper then focuses on whether the bounds from finite horizon research of Neu and Pike-Burke [20] can be applied to these extended value iteration operators in the SSP setting. It shows that similar bounds to [20] for these operators exist, however they lead to operators that are not in general monotone and have more complex convergence properties. In a special case we observe oscillating behaviour. This paper generates open questions on how research may progress, with several examples that require further examination.

show abstract

Section: Stochastic Shortest Paths With Unknown Transitionsmentioning

confidence: 99%

Convex duality for stochastic shortest path problems in known and unknown environments

Francis-Staite¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Proposition 1 ( [21], [22], [24]): The set of all feasible occupancy measures Q is a non-empty polytope given by…”

Section: Preliminaries: Occupancy Measure In Episodic Mdpmentioning

confidence: 99%

“…The system restarts at the end of each episode, and a new episode begins with an initial state. Moreover, following the previous literature [20]- [24], we shall focus on the layered episodic MDP model as described in the following assumption.…”

Section: Problem Formulationmentioning

confidence: 99%

See 1 more Smart Citation

Diffusion of Innovation under Limited-Trust Equilibrium

León

Etesami

Nagi

2022

2022 IEEE 61st Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

We consider online reinforcement learning in episodic Markov decision process (MDP) with an unknown transition matrix and stochastic rewards drawn from a fixed but unknown distribution. The learner aims to learn the optimal policy and minimize their regret over a finite time horizon through interacting with the environment. We devise a simple and efficient model-based algorithm that achieves O(LX √ T A) regret with high probability, where L is the episode length, T is the number of episodes, and X and A are the cardinalities of the state space and the action space, respectively. The proposed algorithm, which is based on the concept of "optimism in the face of uncertainty", maintains confidence sets of transition and reward functions and uses occupancy measures to connect the online MDP with linear programming. It achieves a tighter regret bound compared to the existing works that use a similar confidence sets framework and improves the computational effort compared to those that use a different framework but with a slightly tighter regret bound.

show abstract

“…Related Work Regret minimization in SSP has received much attention recently for both stochastic environment (Tarbouriech et al, 2020;Cohen et al, 2020Cohen et al, , 2021Tarbouriech et al, 2021;Chen et al, 2021a,b;Jafarnia-Jahromi et al, 2021) and adversarial environment (Rosenberg and Mansour, 2021;Chen et al, 2021d;. All previous approaches are either value-based (e.g.…”

Section: Introductionmentioning

confidence: 99%

Policy Optimization for Stochastic Shortest Path

Chen¹,

Luo²,

Rosenberg³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Policy optimization is among the most popular and successful reinforcement learning algorithms, and there is increasing interest in understanding its theoretical guarantees. In this work, we initiate the study of policy optimization for the stochastic shortest path (SSP) problem, a goal-oriented reinforcement learning model that strictly generalizes the finite-horizon model and better captures many applications. We consider a wide range of settings, including stochastic and adversarial environments under full information or bandit feedback, and propose a policy optimization algorithm for each setting that makes use of novel correction terms and/or variants of dilated bonuses . For most settings, our algorithm is shown to achieve a near-optimal regret bound.One key technical contribution of this work is a new approximation scheme to tackle SSP problems that we call stacked discounted approximation and use in all our proposed algorithms. Unlike the finite-horizon approximation that is heavily used in recent SSP algorithms, our new approximation enables us to learn a near-stationary policy with only logarithmic changes during an episode and could lead to an exponential improvement in space complexity.

show abstract

Stochastic Shortest Path with Adversarially Changing Costs

Cited by 7 publications

References 1 publication

Convex duality for stochastic shortest path problems in known and unknown environments

Convex duality for stochastic shortest path problems in known and unknown environments

Diffusion of Innovation under Limited-Trust Equilibrium

Policy Optimization for Stochastic Shortest Path

Contact Info

Product

Resources

About