Distributional Robust Batch Contextual Bandits

Si, Nian; Zhang, Fan; Zhou, Zhengyuan; Blanchet, José

doi:10.48550/arxiv.2006.05630

Cited by 7 publications

(29 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Interestingly, adversarial training can also be understood as solving DRO with a Wasserstein metric-based set [Staib and Jegelka, 2017]. Applications of DRO have been explored in contextual bandits for policy learning [Si et al, 2020, Mo et al, 2020, Faury et al, 2020 and evaluation [Kato et al, 2020, Jeong and. Uncertainty sets based on KL-divergence, L 1 , L 2 and L ∞ norms have been studied [Nilim andEl Ghaoui, 2005, Iyengar, 2005] in robust RL.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Learning Under Adversarial and Interventional Shifts

Singh¹,

Joshi²,

Doshi‐Velez³

et al. 2021

Preprint

View full text Add to dashboard Cite

Machine learning models are often trained on data from one distribution and deployed on others. So it becomes important to design models that are robust to distribution shifts. Most of the existing work focuses on optimizing for either adversarial shifts or interventional shifts. Adversarial methods lack expressivity in representing plausible shifts as they consider shifts to joint distributions in the data. Interventional methods allow more expressivity but provide robustness to unbounded shifts, resulting in overly conservative models. In this work, we combine the complementary strengths of the two approaches and propose a new formulation, RISe, for designing robust models against a set of distribution shifts that are at the intersection of adversarial and interventional shifts. We employ the distributionally robust optimization framework to optimize the resulting objective in both supervised and reinforcement learning settings. Extensive experimentation with synthetic and real world datasets from healthcare demonstrate the efficacy of the proposed approach.

show abstract

Section: Related Workmentioning

confidence: 99%

“…JointDRO accounts for shifts in all variables V and is therefore more conservative than the proposed method. Si et al [2020] recently proposed JointDRO using f -divergences for contextual bandits .…”

Section: Robust Ope In Banditsmentioning

confidence: 99%

Learning Under Adversarial and Interventional Shifts

Singh¹,

Joshi²,

Doshi‐Velez³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The training environment, of course, is unknown, and only data logged by some policy is available. Focusing on a KL-divergence uncertainty set, Si et al [2020] tackle this distributionally robust OPE/L (DROPE/L) problem using an approach based on self-normalized inverse propensity scoring (SNIPS).…”

Section: Introductionmentioning

confidence: 99%

“…One may consider simply fitting and imputing the propensities using some flexible machine learning (ML) algorithms. However, although the SNIPS-based algorithms in Si et al [2020] are robust to environment shifts, they are not robust to propensity estimation errors. As the propensity estimates may converge at slow rates, the SNIPS estimators based on these estimated propensities also tend to converge slowly, leading to slow rates in estimation and learning.…”

Section: Introductionmentioning

confidence: 99%

“…In particular, we show that our objective concentrates uniformly not only over policies, but also over all adversarial environments, yielding our O(N −1/2 ) distributionally robust regret guarantee. Si et al [2020] prove a O(N −1/2 ) distributionally robust regret bound but crucially assumed a known propensity so they can simply estimate the DROPL objective by reweighting. Their proof also used different tricks for discrete and continuous rewards, and is specialized for SNIPS.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Doubly Robust Distributionally Robust Off-Policy Evaluation and Learning

Kallus¹,

Mao²,

Wang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Off-policy evaluation and learning (OPE/L) use offline observational data to make better decisions, which is crucial in applications where experimentation is necessarily limited. OPE/L is nonetheless sensitive to discrepancies between the data-generating environment and that where policies are deployed. Recent work proposed distributionally robust OPE/L (DROPE/L) to remedy this, but the proposal relies on inverse-propensity weighting, whose regret rates may deteriorate if propensities are estimated and whose variance is suboptimal even if not. For vanilla OPE/L, this is solved by doubly robust (DR) methods, but they do not naturally extend to the more complex DROPE/L, which involves a worst-case expectation. In this paper, we propose the first DR algorithms for DROPE/L with KL-divergence uncertainty sets. For evaluation, we propose Localized Doubly Robust DROPE (LDR 2 OPE) and prove its semiparametric efficiency under weak product rates conditions. Notably, thanks to a localization technique, LDR 2 OPE only requires fitting a small number of regressions, just like DR methods for vanilla OPE. For learning, we propose Continuum Doubly Robust DROPL (CDR 2 OPL) and show that, under a product rate condition involving a continuum of regressions, it enjoys a fast regret rate of O(N −1/2 ) even when unknown propensities are nonparametrically estimated. We further extend our results to general f -divergence uncertainty sets. We illustrate the advantage of our algorithms in simulations.

show abstract

Towards Robust Off-Policy Evaluation via Human Inputs

Singh

Joshi

Doshi‐Velez

et al. 2022

Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society

View full text Add to dashboard Cite

Off-policy Evaluation (OPE) methods are crucial tools for evaluating policies in high-stakes domains such as healthcare, where direct deployment is often infeasible, unethical, or expensive. When deployment environments are expected to undergo changes (that is, dataset shifts), it is important for OPE methods to perform robust evaluation of the policies amidst such changes. Existing approaches consider robustness against a large class of shifts that can arbitrarily change any observable property of the environment. This often results in highly pessimistic estimates of the utilities, thereby invalidating policies that might have been useful in deployment. In this work, we address the aforementioned problem by investigating how domain knowledge can help provide more realistic estimates of the utilities of policies. We leverage human inputs on which aspects of the environments may plausibly change, and adapt the OPE methods to only consider shifts on these aspects. Specifically, we propose a novel framework, Robust OPE (ROPE), which considers shifts on a subset of covariates in the data based on user inputs, and estimates worst-case utility under these shifts. We then develop computationally efficient algorithms for OPE that are robust to the aforementioned shifts for contextual bandits and Markov decision processes. We also theoretically analyze the sample complexity of these algorithms. Extensive experimentation with synthetic and real world datasets from the healthcare domain demonstrates that our approach not only captures realistic dataset shifts accurately, but also results in less pessimistic policy evaluations. CCS CONCEPTS• Computing methodologies → Sequential decision making; Adversarial learning; Batch learning; Markov decision processes; Causal reasoning and diagnostics; Model verification and validation.

show abstract

Distributional Robust Batch Contextual Bandits

Cited by 7 publications

References 42 publications

Learning Under Adversarial and Interventional Shifts

Learning Under Adversarial and Interventional Shifts

Doubly Robust Distributionally Robust Off-Policy Evaluation and Learning

Towards Robust Off-Policy Evaluation via Human Inputs

Contact Info

Product

Resources

About