On Learning the Optimal Waiting Time

Lattimore, Tor; György, András; Szepesvári, Csaba

doi:10.1007/978-3-319-11662-4_15

Cited by 20 publications

(23 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A very recent line of research tries to characterize the statistical complexity of interactive decision making, with both upper and lower bounds, based on either the decision-estimation coefficient (DEC) and its variants [FKQR21,FGQ + 22,FRSS22,CMB22,FGH23], or the generalized information ratio [LG21,Lat22]. Although these result typically lead to the right regret dependence on T for general bandit problems, the dependence on d could be loose in both their upper and lower bounds.…”

Section: Related Workmentioning

confidence: 99%

Statistical Complexity and Optimal Algorithms for Non-linear Ridge Bandits

Rajaraman¹,

Jiao²,

Ramchandran³

2023

Preprint

View full text Add to dashboard Cite

We consider the sequential decision-making problem where the mean outcome is a non-linear function of the chosen action. Compared with the linear model, two curious phenomena arise in non-linear models: first, in addition to the "learning phase" with a standard parametric rate for estimation or regret, there is an "burn-in period" with a fixed cost determined by the non-linear function; second, achieving the smallest burn-in cost requires new exploration algorithms. For a special family of non-linear functions named ridge functions in the literature, we derive upper and lower bounds on the optimal burn-in cost, and in addition, on the entire learning trajectory during the burn-in period via differential equations. In particular, a twostage algorithm that first finds a good initial action and then treats the problem as locally linear is statistically optimal. In contrast, several classical algorithms, such as UCB and algorithms relying on regression oracles, are provably suboptimal.

show abstract

Section: Related Workmentioning

confidence: 99%

Statistical Complexity and Optimal Algorithms for Non-linear Ridge Bandits

Rajaraman¹,

Jiao²,

Ramchandran³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…However, the use of Cauchy-Schwarz inequality over the whole action set A is agnostic to the distribution of A * and thus illuminates the effect of a prior. As far as we know, all the existing upper bound analysis of information ratio (Tossou et al, 2017;Lattimore & Szepesvári, 2019;Hao et al, 2021;Hao & Lattimore, 2022) are prior-independent.…”

Section: Why Existing Analysis Is Not Sufficientmentioning

confidence: 99%

Leveraging Demonstrations to Improve Online Learning: Quality Matters

Hao¹,

Jain²,

Lattimore³

et al. 2023

Preprint

View full text Add to dashboard Cite

We investigate the extent to which offline demonstration data can improve online learning. While it is natural to expect some improvement, we show that the degree of improvement depends on the quality of demonstrations. To generate portable insights, we focus on Thompson sampling (TS) applied to a multi-armed bandit as a prototypical online learning algorithm and model. The offline demonstration data is generated by an expert with a given competence level, a notion we introduce. We propose an informed TS algorithm that utilizes demonstration data in a coherent way through Bayes' rule and derive a prior-dependent Bayesian regret bound that offers insight into how conditioning on demonstration data can greatly improve online performance and how the degree of improvement increases with the expert's competence level. We also develop a practical, approximate informed TS algorithm through Bayesian bootstrapping and show substantial empirical regret reduction through experiments.* Equal contribution 1 Deepmind 2 University of Southern California, work done by Rahul Jain while at DeepMind.

show abstract

“…There are other works in the RL literature studying RL problems in the linear setting. [53,33,50] consider the linear setting with a generative model in the time-homogenous case. These works use L ∞ estimation instead of UCB estimation to handle the distribution mismatch phenomenon.…”

Section: Estimation and Optimization Errormentioning

confidence: 99%

Reinforcement Learning with Function Approximation: From Linear to Nonlinear

Long¹,

Han²

2023

Preprint

View full text Add to dashboard Cite

Function approximation has been an indispensable component in modern reinforcement learning algorithms designed to tackle problems with large state space in high dimensions. This paper reviews the recent results on the error analysis of those reinforcement learning algorithms in the settings of linear or nonlinear approximation, with an emphasis on the approximation error and the estimation error/sample complexity. We discuss different properties related to the approximation error and concrete conditions on the transition probability and reward function under which these properties hold true. The sample complexity in reinforcement learning is more complicated for analysis compared to supervised learning, mainly due to the distribution mismatch phenomenon. With assumptions on the linear structure of the problem, there are various algorithms in the literature that can achieve polynomial sample complexity with respect to the number of features, episode length, and accuracy, although the minimax rate has not been achieved yet. These results rely on the L ∞ and UCB estimation of estimation error, which can handle the distribution mismatch phenomenon. The problem and analysis become much more challenging in the setting of nonlinear function approximation since both L ∞ and UCB estimation are inadequate to help bound the error with a good rate in high dimensions. We discuss different additional assumptions needed to handle the distribution mismatch and derive meaningful results for nonlinear RL problems.

show abstract

On Learning the Optimal Waiting Time

Cited by 20 publications

References 11 publications

Statistical Complexity and Optimal Algorithms for Non-linear Ridge Bandits

Statistical Complexity and Optimal Algorithms for Non-linear Ridge Bandits

Leveraging Demonstrations to Improve Online Learning: Quality Matters

Reinforcement Learning with Function Approximation: From Linear to Nonlinear

Contact Info

Product

Resources

About