Whittle index policy for crawling ephemeral content

Abstract

We consider the task of scheduling a crawler to retrieve from several sites their ephemeral content. This is content, such as news or posts at social network groups, for which a user typically loses interest after some days or hours. Thus development of a timely crawling policy for ephemeral information sources is very important. We first formulate this problem as an optimal control problem with average reward. The reward can be measured in terms of the number of clicks or relevant search requests. The problem in its exact formulation suffers from the curse of dimensionality and quickly becomes intractable even with moderate number of information sources. Fortunately, this problem admits a Whittle index, a celebrated heuristics which leads to problem decomposition and to a very simple and efficient crawling policy. We derive the Whittle index and provide its theoretical justification.

A Verification Theorem for Threshold-Indexability of Real-State Discounted Restless Bandits

“…More recent work has addressed indexability in POMDP models with real-state projects, including, e.g., multisite tracking (Le Ny et al (2008)), dynamic multichannel access (Liu and Zhao (2010) and Ouyang et al (2015)), continuous-time (Le Ny et al (2011)) and discrete-time (Dance and Silander (2015)) multitarget tracking, demand response (Taylor and Mathieu (2014)), web crawling (Avrachenkov and Borkar (2018)), and hidden Markov bandits (Meshram et al (2018)).…”

A learning algorithm for the Whittle index policy for scheduling web crawlers

“…where ν i (n) = 1 if location i is crawled at time n and 0 otherwise, subject to the constraint that only N 0 < N crawlers can be active at any time. This can be cast as a restless bandit problem that is Whittle indexable [2]. The Whittle index is given by [2]…”

“…This work addresses a situation when the latter is known in a parametric form, but the parameters are unknown, and the algorithm is required to operate online with streaming real time data. As a test case, we consider the specific problem of scheduling web crawlers for ephemeral content analyzed in [2] (see also [3], [14] for related work). We describe this problem in the next section and summarize the main results of [2].…”

“…As a test case, we consider the specific problem of scheduling web crawlers for ephemeral content analyzed in [2] (see also [3], [14] for related work). We describe this problem in the next section and summarize the main results of [2]. Some preliminary analysis of the problem is given in section III.…”

A Hidden Markov Restless Multi-armed Bandit Model for Playout Recommendation Systems

“…It is solved by the vanishing discount approach [3,26]-by first considering a discounted reward system and then taking limits as the discount approaches to 1. Define…”