Leveraging Demonstrations to Improve Online Learning: Quality Matters

Abstract

We investigate the extent to which offline demonstration data can improve online learning. While it is natural to expect some improvement, we show that the degree of improvement depends on the quality of demonstrations. To generate portable insights, we focus on Thompson sampling (TS) applied to a multi-armed bandit as a prototypical online learning algorithm and model. The offline demonstration data is generated by an expert with a given competence level, a notion we introduce. We propose an informed TS algorithm that utilizes demonstration data in a coherent way through Bayes' rule and derive a prior-dependent Bayesian regret bound that offers insight into how conditioning on demonstration data can greatly improve online performance and how the degree of improvement increases with the expert's competence level. We also develop a practical, approximate informed TS algorithm through Bayesian bootstrapping and show substantial empirical regret reduction through experiments.* Equal contribution 1 Deepmind 2 University of Southern California, work done by Rahul Jain while at DeepMind.