Regret Minimization in Isotonic, Heavy-Tailed Contextual Bandits via Adaptive Confidence Bands

Abstract

In this paper we initiate a study of non parametric contextual bandits under shape constraints on the mean reward function. Specifically, we study a setting where the context is one dimensional, and the mean reward function is isotonic with respect to this context. We propose a policy for this problem and show that it attains minimax rate optimal regret. Moreover, we show that the same policy enjoys automatic adaptation; that is, for subclasses of the parameter space where the true mean reward functions are also piecewise constant with k pieces, this policy remains minimax rate optimal simultaneously for all k ≥ 1. Automatic adaptation phenomena are well-known for shape constrained problems in the offline setting; we show that such phenomena carry over to the online setting. The main technical ingredient underlying our policy is a procedure to derive confidence bands for an underlying isotonic function using the isotonic quantile estimator. The confidence band we propose is valid under heavy tailed noise, and its average width goes to 0 at an adaptively optimal rate. We consider this to be an independent contribution to the isotonic regression literature.

Deep zero-inflated negative binomial model and its application in scRNA-seq data integration

“…Compared to the results in the literature for heavy-tailed bandits without a zero-inflated structure [12,18,14], our regret bound only introduces an additional term,…”

Element-wise Estimation Error of Generalized Fused Lasso

“…A natural question is whether these signal parameters can be estimated which would give a way to obtain finite sample confidence bands for the underlying function. Such elementwise confidence bands are important in application domains such as contextual bandits, see Chatterjee and Sen (2021).…”