Reinforcement Learning, Bit by Bit

Lu, Xiuyuan; Roy, Benjamin Van; Dwaracherla, Vikranth; Ibrahimi, Morteza; Osband, Ian; Wen, Zheng

doi:10.48550/arxiv.2103.04047

Cited by 11 publications

(40 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that this theorem is not the main result of the paper-it serves as a foundation based on which we develop our analysis and a baseline against which we compare our main result. Similar information-theoretic bounds have been established in previous studies, a closely related one being (Lu et al, 2021a).…”

Section: Regret Boundsupporting

confidence: 79%

“…One way of bounding Bayesian regret is through first bounding an agent's information ratio, which is a statistic that quantifies how the agent trades off between regret and information. Various versions of the information ratio have been proposed and studied over the past decade (Russo and Van Roy, 2014a;Bubeck et al, 2015;Russo and Van Roy, 2016;Bubeck and Eldan, 2016;Russo and Van Roy, 2018a;Dong and Van Roy, 2018;Russo and Van Roy, 2018b;Nikolov et al, 2018;Zimmert and Lattimore, 2019;Lattimore and Szepesvári, 2019;Lu and Van Roy, 2019;Bubeck and Sellke, 2020;Lattimore and Szepesvári, 2020;Lattimore and György, 2020;Kirschner et al, 2020;Lu et al, 2021a;Lattimore and Hao, 2021;Devraj et al, 2021). Each depends on beliefs about the environment, as expressed by a prior distribution and likelihood function.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Gaussian Imagination in Bandit Learning

Liu¹,

Devraj²,

Roy³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Assuming distributions are Gaussian often facilitates computations that are otherwise intractable. We study the performance of an agent that attains a bounded information ratio with respect to a bandit environment with a Gaussian prior distribution and a Gaussian likelihood function when applied instead to a Bernoulli bandit. Relative to an information-theoretic bound on the Bayesian regret the agent would incur when interacting with the Gaussian bandit, we bound the increase in regret when the agent interacts with the Bernoulli bandit. If the Gaussian prior distribution and likelihood function are sufficiently diffuse, this increase grows at a rate which is at most linear in the square-root of the time horizon, and thus the per-timestep increase vanishes. Our results formalize the folklore that so-called Bayesian agents remain effective when instantiated with diffuse misspecified distributions.

show abstract

Section: Regret Boundsupporting

confidence: 79%

Section: Introductionmentioning

confidence: 99%

Gaussian Imagination in Bandit Learning

Liu¹,

Devraj²,

Roy³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Information Ratios. Of all the related works described in this section, the work [33] on IDS for MABs discussed above and the recent work [28] developing an information ratio for general problems are most relevant to our current paper. In [28], the authors develop a powerful, abstract framework for reasoning about the design of algorithms for solving sequential decision-making problems, including MDPs.…”

Section: Related Workmentioning

confidence: 99%

“…Prior works in multi-armed bandits and RL [33,28] seek to balance the goals of exploration and exploitation by minimizing the ratio of cost incurred -formulated as regret -to information acquired when determining which action or sequence of actions to choose. See Section 3 for an overview of this line of work.…”

Section: Introductionmentioning

confidence: 99%

“…Our goal is to develop information ratio optimization approaches for infinite-horizon RL problems that can operate in high-dimensional, possibly continuous spaces. Prior works [33,28] define an information ratio in terms of information gain to a time-horizon in the future, which in the infinite-horizon setting requires conditioning over an infinite trajectory. Moreover, tabular value functions are employed, which exhibit scaling proportional to the cardinality of the state and action spaces.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Occupancy Information Ratio: Infinite-Horizon, Information-Directed, Parameterized Policy Search

Suttle¹,

Koppel²,

Liu³

2022

Preprint

View full text Add to dashboard Cite

We develop a new measure of the exploration/exploitation trade-off in infinite-horizon reinforcement learning problems called the occupancy information ratio (OIR), which is comprised of a ratio between the infinite-horizon average cost of a policy and the entropy of its long-term state occupancy measure. The OIR ensures that no matter how many trajectories an RL agent traverses or how well it learns to minimize cost, it maintains a healthy skepticism about its environment, in that it defines an optimal policy which induces a high-entropy occupancy measure. Different from earlier information ratio notions, OIR is amenable to direct policy search over parameterized families, and exhibits hidden quasiconcavity through invocation of the perspective transformation. This feature ensures that under appropriate policy parameterizations, the OIR optimization problem has no spurious stationary points, despite the overall problem's nonconvexity. We develop for the first time policy gradient and actor-critic algorithms for OIR optimization based upon a new entropy gradient theorem, and establish both asymptotic and nonasymptotic convergence results with global optimality guarantees. In experiments, these methodologies outperform several deep RL baselines in problems with sparse rewards, where many trajectories may be uninformative and skepticism about the environment is crucial to success.

show abstract

Subgroup analysis and adaptive experiments crave for debiasing

Wang

2023

WIREs Computational Stats

View full text Add to dashboard Cite

Results obtained from reliably designed randomized experiments are often considered to be evidence of the highest grade for assessing the effectiveness of biomedical or behavioral interventions. Nevertheless, even with randomized experiments, statistical bias can arise in post hoc analysis of the data or through adaptive data collection. In this article, we discuss the need for and review some of the recent developments in statistical methodologies to address the issue of potential bias in adaptive experiments and in subgroup analysis. For adaptive experiments, we focus on adaptive treatment assignments. For subgroup analysis, we focus on post hoc subgroup selection and review several frequentist approaches for debiased inference on the best‐selected subgroup effects.This article is categorized under: Applications of Computational Statistics > Clinical Trials

show abstract

Reinforcement Learning, Bit by Bit

Cited by 11 publications

References 50 publications

Gaussian Imagination in Bandit Learning

Gaussian Imagination in Bandit Learning

Occupancy Information Ratio: Infinite-Horizon, Information-Directed, Parameterized Policy Search

Subgroup analysis and adaptive experiments crave for debiasing

Contact Info

Product

Resources

About