Online Learning via Offline Greedy Algorithms: Applications in Market Design and Optimization

Abstract

Motivated by online decision making in time-varying combinatorial environments, we study the problem of transforming offline algorithms to their online counterparts. We focus on offline combinatorial problems that are amenable to a constant factor approximation using a greedy algorithm that is robust to local errors. For such problems, we provide a general framework that efficiently transforms offline robust greedy algorithms to online ones using Blackwell approachability. We show that the resulting online algorithms have [Formula: see text] (approximate) regret under the full information setting. We further introduce a bandit extension of Blackwell approachability that we call Bandit Blackwell approachability. We leverage this notion to transform greedy robust offline algorithms into a [Formula: see text] (approximate) regret in the bandit setting. Demonstrating the flexibility of our framework, we apply our offline-to-online transformation to several problems at the intersection of revenue management, market design, and online optimization, including product ranking optimization in online platforms, reserve price optimization in auctions, and submodular maximization. We also extend our reduction to greedy-like first-order methods used in continuous optimization, such as those used for maximizing continuous strong DR monotone submodular functions subject to convex constraints. We show that our transformation, when applied to these applications, leads to new regret bounds or improves the current known bounds. We complement our theoretical studies by conducting numerical simulations for two of our applications, in both of which we observe that the numerical performance of our transformations outperforms the theoretical guarantees in practical instances. This paper was accepted by George Shanthikumar, data science.

A Framework for Adapting Offline Algorithms to Solve Combinatorial Multi-Armed Bandit Problems with Bandit Feedback

“…Adversarial CMAB The closest related works are on adversarial CMAB. In (Niazadeh et al, 2021), the authors propose a framework for transforming greedy α-approximation algorithms for offline problems to online methods in an adversarial bandit setting, for both semi-bandit (achieving O(T 1/2 ) α−regret) and full-bandit feedback (achieving O(T 2/3 ) α−regret). Their framework requires the offline approximation algorithm to have an iterative greedy structure (unlike ours), satisfy a robustness property (like ours), and satisfy a property referred to as Blackwell reducibility (unlike ours).…”

“…We also note that (Niazadeh et al, 2021) do not consider submodular CMAB with knapsack constraints, and thus do not verify whether any approximation algorithms for the offline problem satisfy the required properties (of sub-problem structure or robustness or Blackwell reducibility) to be transformed, and this is an example we consider for our general framework. Consequently, in our experiments for submodular CMAB with knapsack constraints in Section 7, we use the algorithm in (Streeter and Golovin, 2008) designed for a knapsack constraint (in expectation) as representative of methods for the adversarial setting.…”

“…Remark 2 In Niazadeh et al (2021), there is a related definition of robustness for offline approximation algorithms. That definition and the subsequent offline-to-online adaptation Algorithm 1 Combinatorial Explore-then-Commit Input: horizon T , set of base elements Ω, an offline (α, δ)-robust algorithm A, and an upper-bound N on the number of A's queries to the value oracle Initialize m ← δ 2/3 T 2/3 log(T ) procedure is restricted to approximation algorithms with an iterative greedy structure.…”

“…Even for the special case of submodular reward functions and a cardinality constraint, it remains an open question. Niazadeh et al (2021) obtain Ω(T 2/3 ) lower bounds for the harder setting where feedback is only available during "exploration" rounds chosen by the agent, who incurs an associated penalty.…”

Product Ranking for Revenue Maximization with Multiple Purchases

“…Many works [5,9,11,10,24] have been proposed to deal with demand learning or price experimentation problems with the help of the framework in the field of revenue management. Recently, more methods [40,17,13,14,6,44] adopt similar techniques in the product ranking setting, which inspires the algorithm proposed in this paper.…”