Hierarchical Adaptive Contextual Bandits for Resource Constraint based Recommendation

Abstract

Contextual multi-armed bandit (MAB) achieves cutting-edge performance on a variety of problems. When it comes to real-world scenarios such as recommendation system and online advertising, however, it is essential to consider the resource consumption of exploration. In practice, there is typically non-zero cost associated with executing a recommendation (arm) in the environment, and hence, the policy should be learned with a fixed exploration cost constraint. It is challenging to learn a global optimal policy directly, since it is a NP-hard problem and significantly complicates the exploration and exploitation trade-off of bandit algorithms. Existing approaches focus on solving the problems by adopting the greedy policy which estimates the expected rewards and costs and uses a greedy selection based on each arm's expected reward/cost ratio using historical observation until the exploration resource is exhausted. However, existing methods are hard to extend to infinite time horizon, since the learning process will be terminated when there is no more resource. In this paper, we propose a hierarchical adaptive contextual bandit method (HATCH) to conduct the policy learning of contextual bandits with a budget constraint. HATCH adopts an adaptive method to allocate the exploration resource based on the remaining resource/time and the estimation of reward distribution among different user contexts. In addition, we utilize full of contextual feature information to find the best personalized recommendation. Finally, in order to prove the theoretical guarantee, we present a regret bound analysis and prove that HATCH achieves a regret bound as low as O( √ T ). The experimental results demonstrate the effectiveness and efficiency of the proposed method on both synthetic data sets and the real-world applications.

Advances and Challenges in Conversational Recommender Systems: A Survey

“…They adopt a generative process based on a topic model to explicitly formulate the arm dependencies as the clusters on arms, where dependent arms are assumed to be generated from the same cluster. Yang et al [197] consider the situations where there are exploration overheads, i.e., there are non-zero costs associated with executing a recommendation (arm) in the environment, and hence, the policy should be learned with a fixed exploration cost constraint. They propose a hierarchical learning structure to address the problem.…”

“…Linear UCB considering item features [92] Considering diversity of recommendation [137,103,40] Cascading bandits providing reliable negative samples [84,230] Combining offline data and online bandit signals [145] Considering pseudo-rewards for arms without feedback [30] Considering dependency among arms [180] Considering exploration overheads [197]…”

A Hybrid Bandit Model with Visual Priors for Creative Ranking in Display Advertising

“…Multi-armed bandits (MAB) problem is a typical sequential decision making process that is also treated as an online decision making problems [32]. A wide range of real world applications can be modeled as MAB problems, such as online recommendation system [16], online advertising [27] and information retrieval [15].…”

Show Me the Whole World

“…For example, ConUCB [40] introduces conversations between the agent and users to ask whether the user is interested in a certain topic occasionally. HATCH [39] considers the resource consumption of exploration and proposes a strategy to conduct bandit exploration with budget limitation. S-MAB [6] considers two aspects, one is to maximize the cumulative rewards and the other is to decide how many arms to be pulled so as to reduce the exploration cost.…”