Improved Optimistic Algorithms for Logistic Bandits

Faury, Louis; Abeille, Marc; Calauzènes, Clément; Fercoq, Olivier

doi:10.48550/arxiv.2002.07530

Cited by 3 publications

(21 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unfortunately, w MLE t may not satisfy Assumption 1, so we instead make use of a projected version of w MLE t . Following Faury et al (2020), and recalling Assumption 3, we define a data matrix and a transformation of w MLE t given by…”

Section: Maximum Likelihood Estimationmentioning

confidence: 99%

“…At each iteration t, we then compute an estimate w L t and determine a set of candidate policies S t for which no other policy π significantly outperforms a member of S t . The threshold for what * A slight modification in the expression of βt(δ) is needed to incorporate the fact that we assume φ(τ ) ≤ B for any τ (Assumption 2) while B = 1 in Faury et al (2020). But this can be easily incorporated using Thm.…”

Section: Algorithm and Analysismentioning

confidence: 99%

“…10 of Abbasi-Yadkori et al (2011) in the final step of the proof of Lem. 12 of Faury et al (2020) constitutes "significant" is determined by the uncertainty in the estimate of w L t . We then search over this set to identify two policies, π 1 t and π 2 t , with expected features that maximize the uncertainty determined by V t , precisely by choosing (π…”

Section: Algorithm and Analysismentioning

confidence: 99%

See 2 more Smart Citations

Dueling RL: Reinforcement Learning with Trajectory Preferences

Pacchiano¹,

Saha²

2021

Preprint

View full text Add to dashboard Cite

We consider the problem of preference based reinforcement learning (PbRL), where, unlike traditional reinforcement learning, an agent receives feedback only in terms of a 1 bit (0/1) preference over a trajectory pair instead of absolute rewards for them. The success of the traditional RL framework crucially relies on the underlying agent-reward model, which, however, depends on how accurately a system designer can express an appropriate reward function and often a non-trivial task. The main novelty of our framework is the ability to learn from preferencebased trajectory feedback that eliminates the need to hand-craft numeric reward models. This paper sets up a formal framework for the PbRL problem with non-markovian rewards, where the trajectory preferences are encoded by a generalized linear model of dimension d. Assuming the transition model is known, we then propose an algorithm with almost optimal regret guarantee of Õ SHd log(T /δ) √ T . We further, extend the above algorithm to the case of unknown transition dynamics, and provide an algorithm with near optimal regret guarantee O((To the best of our knowledge, our work is one of the first to give tight regret guarantees for preference based RL problem with trajectory preferences.

show abstract

Section: Maximum Likelihood Estimationmentioning

confidence: 99%

Section: Algorithm and Analysismentioning

confidence: 99%

Section: Algorithm and Analysismentioning

confidence: 99%

See 1 more Smart Citation

Dueling RL: Reinforcement Learning with Trajectory Preferences

Pacchiano¹,

Saha²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Specifically, logistic bandits, that are appropriate for modeling binary reward structures, are a special case of generalized linear bandits (GLBs) with µ(x) = 1 1+exp(−x) . UCB-based algorithms for GLBs were first introduced in [Filippi et al, 2010, Li et al, 2017, Faury et al, 2020. The same problem, but with a Thompson Sampling-(TS) strategy was also studied in [Abeille et al, 2017, Russo and Van Roy, 2013, Russo and Van Roy, 2014, Dong and Van Roy, 2018.…”

Section: Introductionmentioning

confidence: 99%

“…For this model, we present an algorithm and a corresponding regret bound. Our algorithmic and analytic contribution is in large inspired by very recent exciting progress on binary logistic bandits by [Faury et al, 2020].…”

Section: Introductionmentioning

confidence: 99%

UCB-based Algorithms for Multinomial Logistic Regression Bandits

Amani,

Thrampoulidis

2021

Preprint

View full text Add to dashboard Cite

Out of the rich family of generalized linear bandits, perhaps the most well studied ones are logisitc bandits that are used in problems with binary rewards: for instance, when the learner/agent tries to maximize the profit over a user that can select one of two possible outcomes (e.g., 'click' vs 'no-click'). Despite remarkable recent progress and improved algorithms for logistic bandits, existing works do not address practical situations where the number of outcomes that can be selected by the user is larger than two (e.g., 'click', 'show me later', 'never show again', 'no click'). In this paper, we study such an extension. We use multinomial logit (MNL) to model the probability of each one of K + 1 ≥ 2 possible outcomes (+1 stands for the 'not click' outcome): we assume that for a learner's action xt, the user selects one of K + 1 ≥ 2 outcomes, say outcome i, with a multinomial logit (MNL) probabilistic model with corresponding unknown parameter θ * i. Each outcome i is also associated with a revenue parameter ρi and the goal is to maximize the expected revenue. For this problem, we present MNL-UCB, an upper confidence bound (UCB)-based algorithm, that achieves regret Õ(dK √ T ) with small dependency on problem-dependent constants that can otherwise be arbitrarily large and lead to loose regret bounds. We present numerical simulations that corroborate our theoretical results.

show abstract

Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processes

Zhou¹,

Gu²,

Szepesvári³

2020

Preprint

View full text Add to dashboard Cite

We study reinforcement learning (RL) with linear function approximation where the underlying transition probability kernel of the Markov decision process (MDP) is a linear mixture model (Jia et al., 2020;Ayoub et al., 2020;Zhou et al., 2020) and the learning agent has access to either an integration or a sampling oracle of the individual basis kernels. We propose a new Bernstein-type concentration inequality for self-normalized martingales for linear bandit problems with bounded noise. Based on the new inequality, we propose a new, computationally efficient algorithm with linear function approximation named UCRL-VTR + for the aforementioned linear mixture MDPs in the episodic undiscounted setting. We show that UCRL-VTR + attains an O(dH √ T ) regret where d is the dimension of feature mapping, H is the length of the episode and T is the number of interactions with the MDP. We also prove a matching lower bound Ω(dH √ T ) for this setting, which shows that UCRL-VTR + is minimax optimal up to logarithmic factors. In addition, we propose the UCLK + algorithm for the same family of MDPs under discounting and show that it attains an O(d √ T /(1 − γ) 1.5 ) regret, where γ ∈ [0, 1) is the discount factor. Our upper bound matches the lower bound Ω(d √ T /(1 − γ) 1.5 ) proved in Zhou et al. ( 2020) up to logarithmic factors, suggesting that UCLK + is nearly minimax optimal. To the best of our knowledge, these are the first computationally efficient, nearly minimax optimal algorithms for RL with linear function approximation.

show abstract

Improved Optimistic Algorithms for Logistic Bandits

Cited by 3 publications

References 5 publications

Dueling RL: Reinforcement Learning with Trajectory Preferences

Dueling RL: Reinforcement Learning with Trajectory Preferences

UCB-based Algorithms for Multinomial Logistic Regression Bandits

Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processes

Contact Info

Product

Resources

About