Efficient Counterfactual Learning from Bandit Feedback

Narita, Yoshihiro; Yasui, Shota; Yata, Kohei

doi:10.1609/aaai.v33i01.33014634

Cited by 17 publications

(15 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, in this work, we show that importance sampling with an estimated behavior policy lowers the variance of expectation evaluation in both on-and off-policy settings. Our work complements existing approaches in the causal inference (Rosenbaum 1987;Hirano et al 2003) and bandit (Li et al 2015;Narita et al 2019) literatures that has used importance sampling with an estimated behavior policy as a variance reduction strategy. We extend this general approach to sequential decision making tasks.…”

Section: Introductionmentioning

confidence: 89%

“…For contextual bandits, Narita et al (2019) prove that importance sampling with an estimated behavior policy minimizes asymptotic variance among all asymptotically normal estimators (including ordinary importance sampling). They also provide a large-scale study of policy evaluation with the empirical behavior policy on an adplacement task.…”

Section: Importance Sampling With An Estimated Behavior Policymentioning

confidence: 98%

See 1 more Smart Citation

Importance sampling in reinforcement learning with an estimated behavior policy

2021

View full text Add to dashboard Cite

In reinforcement learning, importance sampling is a widely used method for evaluating an expectation under the distribution of data of one policy when the data has in fact been generated by a different policy. Importance sampling requires computing the likelihood ratio between the action probabilities of a target policy and those of the data-producing behavior policy. In this article, we study importance sampling where the behavior policy action probabilities are replaced by their maximum likelihood estimate of these probabilities under the observed data. We show this general technique reduces variance due to sampling error in Monte Carlo style estimators. We introduce two novel estimators that use this technique to estimate expected values that arise in the RL literature. We find that these general estimators reduce the variance of Monte Carlo sampling methods, leading to faster learning for policy gradient algorithms and more accurate off-policy policy evaluation. We also provide theoretical analysis showing that our new estimators are consistent and have asymptotically lower variance than Monte Carlo estimators.

show abstract

Section: Introductionmentioning

confidence: 89%

Section: Importance Sampling With An Estimated Behavior Policymentioning

confidence: 98%

Importance sampling in reinforcement learning with an estimated behavior policy

2021

View full text Add to dashboard Cite

show abstract

“…Both these approaches do not extend to general function classes and are not applicable to the extreme setting. In the context of off-policy learning from logged data there are several works that address the top-k arms selection problem under the context of slate recommendations (Swaminathan et al, 2017;Narita et al, 2019). We will now review the combinatorial action space literature in multi-armed bandit (MAB) problems.…”

Section: Related Workmentioning

confidence: 99%

Top-$k$ eXtreme Contextual Bandits with Arm Hierarchy

Sen¹,

Rakhlin²,

Ying³

et al. 2021

Preprint

View full text Add to dashboard Cite

Motivated by modern applications, such as online advertisement and recommender systems, we study the top-k eXtreme contextual bandits problem, where the total number of arms can be enormous, and the learner is allowed to select k arms and observe all or some of the rewards for the chosen arms. We first propose an algorithm for the non-eXtreme realizable setting, utilizing the Inverse Gap Weighting strategy for selecting multiple arms. We show that our algorithm has a regret guarantee of O(k (A − k + 1)T log(|F|T )), where A is the total number of arms and F is the class containing the regression function, while only requiring Õ(A) computation per time step. In the eXtreme setting, where the total number of arms can be in the millions, we propose a practically-motivated arm hierarchy model that induces a certain structure in mean rewards to ensure statistical and computational efficiency. The hierarchical structure allows for an exponential reduction in the number of relevant arms for each context, thus resulting in a regret guarantee of O(k (log A − k + 1)T log(|F|T )). Finally, we implement our algorithm using a hierarchical linear function class and show superior performance with respect to wellknown benchmarks on simulated bandit feedback experiments using eXtreme multi-label classification datasets. On a dataset with three million arms, our reduction scheme has an average inference time of only 7.9 milliseconds, which is a 100x improvement.

show abstract

“…For example, Hanna et al [2021] show how importance sampling with an estimated behavior policy can lower sampling error and lead to more accurate policy evaluation. Similar methods have also been studied for policy evaluation in multi-armed bandits Li et al [2015], Narita et al [2019], temporal-difference learning Pavse et al [2020], and policy gradient RL Hanna et al [2021]. All of these prior works assume that data is available a priori and ignore the question of how to collect data if it is unavailable.…”

Section: Related Workmentioning

confidence: 99%

Robust On-Policy Data Collection for Data-Efficient Policy Evaluation

Zhong¹,

Zhang²,

Schäfer³

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper considers how to complement offline reinforcement learning (RL) data with additional data collection for the task of policy evaluation. In policy evaluation, the task is to estimate the expected return of an evaluation policy on an environment of interest. Prior work on offline policy evaluation typically only considers a static dataset. We consider a setting where we can collect a small amount of additional data to combine with a potentially larger offline RL dataset. We show that simply running the evaluation policy -on-policy data collection -is sub-optimal for this setting. We then introduce two new data collection strategies for policy evaluation, both of which consider previously collected data when collecting future data so as to reduce distribution shift (or sampling error) in the entire dataset collected. Our empirical results show that compared to on-policy sampling, our strategies produce data with lower sampling error and generally lead to lower mean-squared error in policy evaluation for any total dataset size. We also show that these strategies can start from initial off-policy data, collect additional data, and then use both the initial and new data to produce low mean-squared error policy evaluation without using off-policy corrections.In many RL applications, data-efficient policy evaluation is of the upmost importance -we desire the most accurate estimate with minimal collected data. Towards data-efficient policy evaluation, much research has gone into how to most efficiently use a set of already collected data [Precup et al., 2000, Thomas andBrunskill, 2016]. Comparatively less research has considered the question of how to improve data collection for the data-efficiency of policy evaluation. This paper focuses on this latter question.

show abstract

Efficient Counterfactual Learning from Bandit Feedback

Cited by 17 publications

References 13 publications

Importance sampling in reinforcement learning with an estimated behavior policy

Importance sampling in reinforcement learning with an estimated behavior policy

Top-$k$ eXtreme Contextual Bandits with Arm Hierarchy

Robust On-Policy Data Collection for Data-Efficient Policy Evaluation

Contact Info

Product

Resources

About