Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Bounds

Zanette, Andrea; Brunskill, Emma

doi:10.48550/arxiv.1901.00210

Cited by 15 publications

(39 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Though there has been a fair amount of research for preference based bandits (no state information), only very few works consider incorporating preference feedback in the reinforcement learning (RL) framework that optimizes the accumulated long-term reward of a suitably chosen reward function over a markov decision process (Singh et al, 2002;Ng et al, 2006;Talebi and Maillard, 2018;Ortner, 2020;Zhang and Ji, 2019;Zanette and Brunskill, 2019). However the classical RL setup considers accessibility of reward feedback per state-action pair whereas which might be impractical in many real world scenarios.…”

Section: Related Workmentioning

confidence: 99%

Dueling RL: Reinforcement Learning with Trajectory Preferences

Pacchiano¹,

Saha²

2021

Preprint

View full text Add to dashboard Cite

We consider the problem of preference based reinforcement learning (PbRL), where, unlike traditional reinforcement learning, an agent receives feedback only in terms of a 1 bit (0/1) preference over a trajectory pair instead of absolute rewards for them. The success of the traditional RL framework crucially relies on the underlying agent-reward model, which, however, depends on how accurately a system designer can express an appropriate reward function and often a non-trivial task. The main novelty of our framework is the ability to learn from preferencebased trajectory feedback that eliminates the need to hand-craft numeric reward models. This paper sets up a formal framework for the PbRL problem with non-markovian rewards, where the trajectory preferences are encoded by a generalized linear model of dimension d. Assuming the transition model is known, we then propose an algorithm with almost optimal regret guarantee of Õ SHd log(T /δ) √ T . We further, extend the above algorithm to the case of unknown transition dynamics, and provide an algorithm with near optimal regret guarantee O((To the best of our knowledge, our work is one of the first to give tight regret guarantees for preference based RL problem with trajectory preferences.

show abstract

Section: Related Workmentioning

confidence: 99%

Dueling RL: Reinforcement Learning with Trajectory Preferences

Pacchiano¹,

Saha²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…There has been a long line of research studying Markov Decision Process (MDP), which can be viewed as a single-agent version of Markov Game. Tabular MDP has been studied thoroughly in recent years [7,22,14,2,60,25,64]. Particularly, in the episodic setting, the minimax regret or sample complexity is achieved by both model-based [2] and model-free [64] methods, up to logarithmic factors.…”

Section: Introductionmentioning

confidence: 99%

The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces

Jin,

Liu,

2021

Preprint

View full text Add to dashboard Cite

Modern reinforcement learning (RL) commonly engages practical problems with large state spaces, where function approximation must be deployed to approximate either the value function or the policy. While recent progresses in RL theory address a rich set of RL problems with general function approximation, such successes are mostly restricted to the single-agent setting. It remains elusive how to extend these results to multi-agent RL, especially in the face of new game-theoretical challenges. This paper considers two-player zero-sum Markov Games (MGs). We propose a new algorithm that can provably find the Nash equilibrium policy using a polynomial number of samples, for any MG with low multi-agent Bellman-Eluder dimension-a new complexity measure adapted from its single-agent version [26]. A key component of our new algorithm is the exploiter, which facilitates the learning of the main player by deliberately exploiting her weakness. Our theoretical framework is generic, which applies to a wide range of models including but not limited to tabular MGs, MGs with linear or kernel function approximation, and MGs with rich observations.

show abstract

“…Therefore, a natural extension is to devise an algorithm that can adapt to an unknown sparsity level. Moreover, it is well known that structural assumptions can lead to improved regret bounds, and previous works proposed algorithms whose regret depends on nontrivial structural properties of the problem (Maillard et al, 2014;Zanette & Brunskill, 2019;Foster et al, 2019;. Thus, it is interesting to understand what structural properties (beyond sparsity) affect the budgeted performance and how to design algorithms that adapt to such properties.…”

Section: Summary and Discussionmentioning

confidence: 99%

“…While CBM-UCBVI clearly demonstrates the analysis techniques and insights from applying the CBM principle to RL, it is of interest to combine it with an algorithm with order-optimal regret bounds of √ SAH 3 T (e.g., (Jin et al, 2018)) when B(t) = Ht, that is, in the standard RL setting (notice that T is the number of episodes and not the total number of time steps). We achieve this goal by performing a more refined analysis that uses tighter concentration results based on (Azar et al, 2017;Dann et al, 2019;Zanette & Brunskill, 2019). Indeed, doing so leads to tighter regret bounds by a √ H factor in the leading term (Full details on the algorithm and proofs can be found at Appendix F).…”

Section: Reinforcement Learningmentioning

confidence: 99%

“…Finally, the environment changes its state based on the agent's action, and the cycle begins anew. Much effort had been devoted to study specific instances of this abstract model, e.g., multi-armed bandits (MABs) (Auer et al, 2002;Garivier & Cappé, 2011;Kaufmann et al, 2012;Agrawal & Goyal, 2012), linear bandits (Dani et al, 2008;Abbasi-Yadkori et al, 2011;Agrawal & Goyal, 2013;Abeille et al, 2017) and reinforcement learning (RL) settings (Azar et al, 2017;Jin et al, 2018;Dann et al, 2019;Zanette & Brunskill, 2019;Efroni et al, 2019;Simchowitz & Jamieson, 2019;Tarbouriech et al, 2020;Cohen et al, 2020;Zhang et al, 2020). However, there are (still) several gaps between theory and practice that hinder the application of these models in real-world problems.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Confidence-Budget Matching for Sequential Budgeted Learning

Efroni,

Merlis,

Saha

et al. 2021

Preprint

View full text Add to dashboard Cite

A core element in decision-making under uncertainty is the feedback on the quality of the performed actions. However, in many applications, such feedback is restricted. For example, in recommendation systems, repeatedly asking the user to provide feedback on the quality of recommendations will annoy them. In this work, we formalize decision-making problems with querying budget, where there is a (possibly time-dependent) hard limit on the number of reward queries allowed. Specifically, we consider multi-armed bandits, linear bandits, and reinforcement learning problems. We start by analyzing the performance of 'greedy' algorithms that query a reward whenever they can. We show that in fully stochastic settings, doing so performs surprisingly well, but in the presence of any adversity, this might lead to linear regret. To overcome this issue, we propose the Confidence-Budget Matching (CBM) principle that queries rewards when the confidence intervals are wider than the inverse square root of the available budget. We analyze the performance of CBM based algorithms in different settings and show that they perform well in the presence of adversity in the contexts, initial states, and budgets. * Equal contribution, alphabetical order 1 Microsoft Research, New York 2 Technion, Israel.

show abstract

Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Bounds

Cited by 15 publications

References 3 publications

Dueling RL: Reinforcement Learning with Trajectory Preferences

Dueling RL: Reinforcement Learning with Trajectory Preferences

The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces

Confidence-Budget Matching for Sequential Budgeted Learning

Contact Info

Product

Resources

About