On Incomplete Learning and Certainty-Equivalence Control

Keskin, N. Bora; Zeevi, Assaf

doi:10.1287/opre.2017.1713

Cited by 40 publications

(7 citation statements)

References 39 publications

(47 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This means that it suffices to analyse the second term which is the (expected) regret that has been analysed in [5,16] assuming a self-exploration property of greedy policies (see Remark 3.1). However, the following examples shows these greedy policies in general do not guarantee exploration and consequently convergence to the optimal solution, which is often referred to as incomplete learning in the literature (see e.g., [21]).…”

Section: Phased-based Learning Algorithm and Our Contributionsmentioning

confidence: 99%

“…They prove that if optimal controls of the true model automatically exploit the parameter space, then a greedy least-squares algorithm with suitable initialisation admits a non-asymptotically logarithmic expected regret for LQ models [5], and a O( √ N ) expected regret for LC models [16]. Unfortunately, as shown in Example 1.1 and in [21], such a self-exploration property may not hold in general, even for LQ models. Furthermore, the learning algorithm studied here works for an arbitrary initialisation.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models

Szpruch¹,

Treetanthiploet²,

Zhang³

2021

Preprint

View full text Add to dashboard Cite

We develop a probabilistic framework for analysing model-based reinforcement learning in the episodic setting. We then apply it to study finite-time horizon stochastic control problems with linear dynamics but unknown coefficients and convex, but possibly irregular, objective function. Using probabilistic representations, we study regularity of the associated cost functions and establish precise estimates for the performance gap between applying optimal feedback control derived from estimated and true model parameters. We identify conditions under which this performance gap is quadratic, improving the linear performance gap in recent work [X. Guo, A. Hu, and Y. Zhang, arXiv preprint, arXiv:2104.09311, (2021)], which matches the results obtained for stochastic linear-quadratic problems. Next, we propose a phase-based learning algorithm for which we show how to optimise exploration-exploitation trade-off and achieve sublinear regrets in high probability and expectation. When assumptions needed for the quadratic performance gap hold, the algorithm achieves an order O( √ N ln N ) high probability regret, in the general case, and an order O((ln N ) 2 ) expected regret, in self-exploration case, over N episodes, matching the best possible results from the literature. The analysis requires novel concentration inequalities for correlated continuous-time observations, which we derive.

show abstract

Section: Phased-based Learning Algorithm and Our Contributionsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models

Szpruch¹,

Treetanthiploet²,

Zhang³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…More recently, some have studied dynamic pricing with changing demand covariates (Cohen et al 2016, Qiang and Bayati 2016, Javanmard and Nazerzadeh 2016, Ban and Keskin 2017 or a changing demand function (den Boer 2015, Keskin andZeevi 2015). These changes in the demand environment can help the greedy algorithm explore naturally and achieve asymptotically optimal performance.…”

Section: Related Literaturementioning

confidence: 99%

Mostly Exploration-Free Algorithms for Contextual Bandits

Bastani¹,

Bayati²,

Khosravi³

2017

Preprint

View full text Add to dashboard Cite

The contextual bandit literature has traditionally focused on algorithms that address the explorationexploitation tradeoff. In particular, greedy algorithms that exploit current estimates without any exploration may be sub-optimal in general. However, exploration-free greedy algorithms are desirable in practical settings where exploration may be costly or unethical (e.g., clinical trials). Surprisingly, we find that a simple greedy algorithm can be rate-optimal (achieves asymptotically optimal regret) if there is sufficient randomness in the observed contexts (covariates). We prove that this is always the case for a two-armed bandit under a general class of context distributions that satisfy a condition we term covariate diversity. Furthermore, even absent this condition, we show that a greedy algorithm can be rate optimal with positive probability. Thus, standard bandit algorithms may unnecessarily explore. Motivated by these results, we introduce Greedy-First, a new algorithm that uses only observed contexts and rewards to determine whether to follow a greedy algorithm or to explore. We prove that this algorithm is rate-optimal without any additional assumptions on the context distribution or the number of arms. Extensive simulations demonstrate that Greedy-First successfully reduces exploration and outperforms existing (exploration-based) contextual bandit algorithms such as Thompson sampling or upper confidence bound (UCB).

show abstract

“…In our numerical analysis (see Section 7) we also include a lookahead type policy. Some recent work have highlighted the effectiveness of simple policies in the context of dynamic learning such as greedy algorithms (e.g., Bastani et al, 2020) and Certainty-Equivalence (e.g., Keskin and Zeevi, 2018). The greedy algorithm behaves well when exploration is expensive while in our case it is free.…”

Section: Related Literaturementioning

confidence: 87%

Diffusion Approximations for a Class of Sequential Testing Problems

Araman¹,

Caldentey²

2021

Preprint

View full text Add to dashboard Cite

We consider a decision maker who must choose an action in order to maximize a reward function that depends on the action that she selects as well as on an unknown parameter Θ. The decision maker can delay taking the action in order to experiment and gather additional information on Θ. We model the decision maker's problem using a Bayesian sequential experimentation framework and use dynamic programming and diffusion-asymptotic analysis to solve it. For that, we scale our problem in a way that both the average number of experiments that is conducted per unit of time is large and the informativeness of each individual experiment is low. Under such regime, we derive a diffusion approximation for the sequential experimentation problem, which provides a number of important insights about the nature of the problem and its solution. First, it reveals that the problems of (i) selecting the optimal sequence of experiments to use and (ii) deciding the optimal time when to stop experimenting decouple and can be solved independently. Second, it shows that an optimal experimentation policy is one that chooses the experiment that maximizes the instantaneous volatility of the belief process. Third, the diffusion approximation provides a more mathematically malleable formulation that we can solve in closed form and suggests efficient heuristics for the non-asympototic regime. Our solution method also shows that the complexity of the problem grows only quadratically with the cardinality of the set of actions from which the decision maker can choose.We illustrate our methodology and results using a concrete application in the context of assortment selection and new product introduction. Specifically, we study the problem of a seller who wants to select an optimal assortment of products to launch into the marketplace and is uncertain about consumers' preferences. Motivated by emerging practices in e-commerce, we assume that the seller is able to use a crowdvoting system to learn these preferences before a final assortment decision is made. In this context, we undertake an extensive numerical analysis to assess the value of learning and demonstrate the effectiveness and robustness of the heuristics derived from the diffusion approximation.

show abstract

On Incomplete Learning and Certainty-Equivalence Control

Cited by 40 publications

References 39 publications

Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models

Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models

Mostly Exploration-Free Algorithms for Contextual Bandits

Diffusion Approximations for a Class of Sequential Testing Problems

Contact Info

Product

Resources

About