Optimal Learning for Stochastic Optimization with Nonlinear Parametric Belief Models

Abstract

We consider the problem of estimating the expected value of information (the knowledge gradient) for Bayesian learning problems where the belief model is nonlinear in the parameters. Our goal is to maximize some metric, while simultaneously learning the unknown parameters of the nonlinear belief model, by guiding a sequential experimentation process which is expensive. We overcome the problem of computing the expected value of an experiment, which is computationally intractable, by using a sampled approximation, which helps to guide experiments but does not provide an accurate estimate of the unknown parameters. We then introduce a resampling process which allows the sampled model to adapt to new information, exploiting past experiments. We show theoretically that the method converges asymptotically to the true parameters, while simultaneously maximizing our metric. We show empirically that the process exhibits rapid convergence, yielding good results with a very small number of experiments.

Reinforcement Learning for Dynamic Bidding in Truckload Markets: an Application to Large-Scale Fleet Management with Advance Commitments

“…This can be a reasonable approximation in one or two dimensions, but tends to become quite poor in three or more dimensions. He and Powell (2018) relaxes the restriction of a fixed set of θ's by adaptively resampling a large pool and choosing new values based on mean squared error. While this logic was shown to be useful for higher dimensional problems (e.g.…”

“…In Section 4.2, we provide the first asymptotic convergence theorem for the contextual knowledge gradient policy. In Section 4.3, we design a resampling procedure specifically for convex loss functions using bootstrap aggregation to find the most promising parameter values, without the need to maintain or enumerate a large pool as used in He and Powell (2018).…”