Statistical Inference for Online Decision Making: In a Contextual Bandit Setting

Abstract

Online decision-making problem requires us to make a sequence of decisions based on incremental information. Common solutions often need to learn a reward model of different actions given the contextual information and then maximize the long-term reward. It is meaningful to know if the posited model is reasonable and how the model performs in the asymptotic sense. We study this problem under the setup of the contextual bandit framework with a linear reward model. The ε-greedy policy is adopted to address the classic exploration-and-exploitation dilemma. Using the martingale central limit theorem, we show that the online ordinary least squares estimator of model parameters is asymptotically normal. When the linear model is misspecified, we propose the online weighted least squares estimator using the inverse propensity score weighting and also establish its asymptotic normality. Based on the properties of the parameter estimators, we further show that the in-sample inverse propensity weighted value estimator is asymptotically normal. We illustrate our results using simulations and an application to a news article recommendation dataset from Yahoo!.

Doubly Robust Interval Estimation for Optimal Policy Evaluation in Online Learning

“…Since the optimal policy is unknown, we estimate the optimal policy from the online data as π t , to infer the value of interest. As commonly assumed in the current online inference literature (see e.g., Deshpande et al, 2018;Zhang et al, 2020;Chen et al, 2020) and the bandit literature (see e.g., Chu et al, 2011;Abbasi-Yadkori et al, 2011;Bubeck and Cesa-Bianchi, 2012;Zhou, 2015), we consider the conditional mean outcome function takes a linear form, i.e., µ(x, a) = x β(a), where β(•) is a smooth function, which can be estimated via a ridge regression based on H t−1 as…”

“…as the number of pulls for action a, y t−1 (a) is the N t−1 (a)×1 vector of the outcomes received under action a at time t − 1, and ω is the regularization term. There are two main reasons to choose the ridge estimator instead of the ordinary least square estimator that is considered in Deshpande et al (2018); Zhang et al (2020); Chen et al (2020). First, the ridge estimator is well defined when D t−1 (a) D t−1 (a) is singular, and its bias is negligible when the time step is large.…”

“…We establish the relationship between p t and κ t in the next section and discuss when assumption 3.1 cannot hold. Assumption 3.2 is a technical condition on bounded contexts such that the mean of the martingale differences will converge to zero (see e.g., Zhang et al, 2020;Chen et al, 2020).…”

“…Chambaz et al (2017) established the asymptotic normality for the conditional mean outcome under an optimal policy for sequential decision making. Another work in Chen et al (2020) proposed an inverse probability weighted value estimator to infer the value of optimal policy using the EG method. These two works did not discuss how to account for the exploration-and-exploitation trade-off under commonly used bandit algorithms, such as UCB and TS considered in this paper.…”

Statistical Inference for Online Decision Making via Stochastic Gradient Descent

“…They used ε-greedy method with ε being a function of the estimated treatment effect and gave the asymptotic distribution of the mean reward under the optimal decision rule. Chen, Lu, and Song (2020) studied the contextual bandit problem with a linear reward model and also adopted ε-greedy method but their ε is a function of decision steps. They gave the asymptotic distributions for the reward model parameters and the expected reward under the optimal decision rule.…”