Integrated Online Learning and Adaptive Control in Queueing Systems with Uncertain Payoffs

Abstract

Many online service platforms have dedicated algorithms to match their available resources to incoming clients to maximize client satisfaction. One of the key challenges is to balance the generation of higher payoffs from existing clients and exploration of new clients’ unknown characteristics while at the same time satisfy the resource capacity constraints. In “Integrated Online Learning and Adaptive Control in Queueing Systems with Uncertain Payoffs,” Hsu, Xu, Lin, and Bell show that traditional approaches such as maximizing instantaneous payoffs with current knowledge or using queue-length based controls guided by “shadow prices,” would lead to suboptimal long-term payoffs. Instead, they propose a novel utility-guided assignment algorithm that seamlessly integrates online learning and adaptive control to provide high system payoffs with performance guarantees. The theoretical performance bound also lends system design insights into the impact of uncertain client dynamics, payoff learning, and backlogged clients. They further develop a decentralized version of the algorithm, which is applicable to large systems and performs well even when the service rates are random.

Efficient Learning-based Scheduling for Information Freshness in Wireless Networks

“…Then, we carefully incorporate the regret analysis for classical UCB algorithm (e.g., [5]) into our analysis. The analysis is similar to the line of regret analysis in [28] and [12], and is available in Appendix B.…”

Scheduling Servers with Stochastic Bilinear Rewards

“…Our algorithm randomly assigns jobs to servers in each time step with the expected number of assignments of jobs of a class to servers of a class determined by the solution of a system optimization problem, whose objective function consists of a fairness of allocation term and a reward maximization term. This dynamic allocation method follows the framework proposed in Hsu et al (2021) with the system optimization problem definition following the utility maximization framework studied in the context of network resource allocation. In our algorithm, the reward maximization part of the system optimization problem objective uses mean reward estimates computed by following an optimism in the face of uncertainty strategy developed for linear bandits in Abbasi-Yadkori et al (2011), which we adapted to our setting.…”

“…A closely related queueing system control problem was studied in Hsu et al (2021) with key differences in that we consider systems where the scheduler has access to features of jobs and servers and rewards of job-server assignments follow a bilinear model, and we also consider more general cases in which mean job service times are allowed to be different across job classes and the set of server classes can be time varying. The bilinear structure of rewards allows us to design algorithms that extend learning to be over job classes, which is different from the approach in Hsu et al (2021) where the learning is for each job separately. As a result, we obtain better regret bounds which scale sub-linearly rather than linearly with the time horizon T and remove the dependence on the number of server classes J by exploiting the structure of rewards.…”

“…We use the framework for regret analysis proposed in Hsu et al (2021) et al (2021). The goal of these works is to derive better regret bounds by exploiting the low-rank structure of the unknown parameter-in particular, to improve the dependency on dimension parameter d. Bilinear bandit problems were also studied under a graph structure (Rizk et al 2021) where feature vectors represent vertices of a graph and a bilinear model is used to model edges.…”

“…The design of the algorithm follows the utility-maximization framework introduced in Hsu et al (2021), which is extended to combine scheduling with learning using linear bandit algorithms in Abbasi-Yadkori et al (2011), Qin et al (2014). The utility-maximization framework accounts for fairness of server Algorithm 1 Utility maximization with bilinear rewards…”