Cost-aware Cascading Bandits

Abstract

In this paper, we propose a cost-aware cascading bandits model, a new variant of multi-armed bandits with cascading feedback, by considering the random cost of pulling arms. In each step, the learning agent chooses an ordered list of items and examines them sequentially, until certain stopping condition is satisfied. Our objective is then to maximize the expected net reward in each step, i.e., the reward obtained in each step minus the total cost incurred in examining the items, by deciding the ordered list of items, as well as when to stop examination. We study both the offline and online settings, depending on whether the state and cost statistics of the items are known beforehand. For the offline setting, we show that the Unit Cost Ranking with Threshold 1 (UCR-T1) policy is optimal. For the online setting, we propose a Cost-aware Cascading Upper Confidence Bound (CC-UCB) algorithm, and show that the cumulative regret scales in O(log T ). We also provide a lower bound for all α-consistent policies, which scales in Ω(log T ) and matches our upper bound. The performance of the CC-UCB algorithm is evaluated with both synthetic and real-world data.

Neighbor Cell List Optimization in Handover Management Using Cascading Bandits Algorithm

“…Lastly, the configuration of eNCL is done online, via iterations with UE handovers. This naturally poses as a bandit problem, therefore we leverage the cost-aware cascading bandits [26] in the algorithm design. However, unlike the simple cascading bandits setting in [26], UDN based on small BSs (such as femto and pico), often encounter delay or missing UE measurements, based on which we have modeled the features of reward and cost on the proposed algorithm.…”

“…To achieve more accurate evaluation of BSs, we take both channel quality and cell capacity of BSs into consideration in NCL optimization. In order to explicitly enforce the constraint of delay in handover preparation phase, we have modified the cost-aware cascading bandits framework proposed in [26] and modelled the number of scanned BSs as a random cost.…”

“…In each step, the policy uses it as an estimation for the corresponding reward expectation, and picks the item with the current highest index for the next play [30]. One critical limitation of applying the algorithm proposed in [26] to handover management, is that handover delay could not be taken into consideration. We are interested in finding a NCL configuration algorithm that minimize signaling cost and handover delay.…”

Observe Before Play: Multi-Armed Bandit with Pre-Observations

“…The proposed MAB with pre-observations in a singleplayer setting is a variant on cascading bandits (Kveton et al 2015a;2015b;Zong et al 2016). The idea of preobservations with costs is similar to the cost-aware cascading bandits proposed in (Zhou et al 2018) and contextual combinatorial cascading bandits introduced in (Li et al 2016). However, in (Zhou et al 2018), the reward collected by the player can be negative if all selected arms have zero reward in one round; in our model, the player will get zero reward if all selected arms are unavailable.…”

“…The idea of preobservations with costs is similar to the cost-aware cascading bandits proposed in (Zhou et al 2018) and contextual combinatorial cascading bandits introduced in (Li et al 2016). However, in (Zhou et al 2018), the reward collected by the player can be negative if all selected arms have zero reward in one round; in our model, the player will get zero reward if all selected arms are unavailable. Moreover, most cascading bandit algorithms are applied to recommendation systems, where there is only a single player.…”