Non-Stationary Representation Learning in Sequential Linear Bandits

Abstract

In this paper, we study representation learning for multi-task decision-making in non-stationary environments. We consider the framework of sequential linear bandits, where the agent performs a series of tasks drawn from distinct sets associated with different environments. The embeddings of tasks in each set share a low-dimensional feature extractor called representation, and representations are different across sets. We propose an online algorithm that facilitates efficient decision-making by learning and transferring non-stationary representations in an adaptive fashion. We prove that our algorithm significantly outperforms the existing ones that treat tasks independently. We also conduct experiments using both synthetic and real data to validate our theoretical insights and demonstrate the efficacy of our algorithm.

Nearly Minimax Algorithms for Linear Bandits with Shared Representation

“…Meta-learning is closely related to representation learning, although the assumption is not the same [Denevi et al, 2019;Finn et al, 2019;Lee et al, 2019;Bertinetto et al, 2018]. There are also other works on different directions such linear MDP with a generative model [Lu et al, 2021], non-stationary sequential learning [Qin et al, 2022], and linear dynamical systems [Modi et al, 2021].…”

Representation Learning for Context-Dependent Decision-Making

“…since η t is an independent random variable with zero mean. It can be derived (more details can be found in the extended version of this paper [29]) that E s 1 2 ≤ 2cφ 2 max ε 2 for some constant c and E s…”

“…Proof of Theorem 3.5: After the RE phase of nth cycle in the SeqRepL algorithm, it can be derived (more details can be found in the extended version of this paper [29]) that the estimate B and the true representation…”