Low-Rank Generalized Linear Bandit Problems

Abstract

In a low-rank linear bandit problem, the reward of an action (represented by a matrix of size d1 × d2) is the inner product between the action and an unknown low-rank matrix Θ * . We propose an algorithm based on a novel combination of online-to-confidence-set conversion (Abbasi-Yadkori et al., 2012) and the exponentially weighted average forecaster constructed by a covering of low-rank matrices. In T rounds, our algorithm achieves O((d1 + d2) 3/2 √ rT ) regret that improves upon the standard linear bandit regret bound of O(d1d2 √ T ) when the rank of Θ * : r min{d1, d2}. We also extend our algorithmic approach to the generalized linear setting to get an algorithm which enjoys a similar bound under regularity conditions on the link function. To get around the computational intractability of covering based approaches, we propose an efficient algorithm by extending the "Explore-Subspace-Then-Refine" algorithm of Jun et al. (2019). Our efficient algorithm achieves O((d1 + d2) 3/2 √ rT ) regret under a mild condition on the action set X and the r-th singular value of Θ * . Our upper bounds match the conjectured lower bound of Jun et al. ( 2019) for a subclass of low-rank linear bandit problems. Further, we show that existing lower bounds for the sparse linear bandit problem strongly suggest that our regret bounds are unimprovable. To complement our theoretical contributions, we also conduct experiments to demonstrate that our algorithm can greatly outperform the performance of the standard linear bandit approach when Θ * is low-rank.

Nearly Minimax Algorithms for Linear Bandits with Shared Representation

“…Our setting is also related to the recent line of work on low-rank bandits [Lale et al, 2019;Lu et al, 2020;Jun et al, 2019;Lattimore and Hao, 2021;, because our formulation also admits a low-rank structure. However, the approach we use and their are very different.…”

Near-optimal Representation Learning for Linear Bandits and Linear RL

“…The mean reward in their setting is defined as the bilinear multiplication x ⊤ Θy, where x and y are two actions selected at each step, and Θ is an unknown parameter matrix with low rank. Their setting is further generalized by Lu et al (2020). Furthermore, sparse linear bandits can be regarded as a simplified setting, where B is a binary matrix indicating the subset of relevant features in context x (Abbasi- Yadkori et al, 2012;Carpentier and Munos, 2012;Lattimore et al, 2015;Hao et al, 2020).…”

Impact of Representation Learning in Linear Bandits

“…Their proposed algorithm shares some similarity as our algorithm for the infinite-action setting in that they added an exploration stage to extract the low-rank structure of Θ. Their setting is further generalized and studied by Lu et al [2020].…”