Non-stationary Continuum-armed Bandits for Online Hyperparameter Optimization

Abstract

For years, machine learning has become the dominant approach to a variety of information retrieval tasks. The performance of machine learning algorithms heavily depends on their hyperparameters. It is hence critical to identity the optimal hyperparameter configuration when applying machine learning algorithms. Most of existing hyperparameter optimization methods assume a static relationship between hyperparameter configuration and algorithmic performance and are thus not suitable for many information retrieval applications with non-stationary environments such as ecommerce recommendation and online advertising. To address this limitation, we study online hyperparameter optimization, where the hyperparameter configuration is optimized on the fly. We formulate online hyperparameter optimization as a non-stationary continuum-armed bandits problem in which each arm corresponds to a hyperparameter configuration and the algorithmic performance is viewed as reward. For this problem, we develop principled methods with strong theoretical guarantees in terms of dynamic regret. The key idea is to adaptively discretize the continuous arm set and estimate the mean reward of each arm via weighted averaging. As a case application, we show how our methods can be applied to optimize the hyperparameter of vector-based candidate generation algorithm and empirically demonstrate the effectiveness and efficiency of our methods on public advertising dataset and online

Contextual Multi-Armed Bandit With Costly Feature Observation in Non-Stationary Environments

“…Non-stationary multi-armed bandits have attracted intensive attention in the past years, both from the theory [9]- [13] and the application [14]- [17] side. Potential application domains span across different fields, including online recommender systems [3], [15]- [17], hyperparameter optimization [18], virtual reality for rehabilitation [19], split liver transplantation allocation [20], evaluation of information retrieval systems [21], or targeted Covid-19 border testing of travelers [22]. The state-of-the-art methods in non-stationary bandits either do not consider access to contextual information or do not assume costly information acquisition.…”

Managing Considerable Distributed Resources for Demand Response: A Resource Selection Strategy Based on Contextual Bandit

“…Currently, many efforts have been devoted to dealing with sequential decision-making problems using bandit-based approaches, where the decision maker seeks to select the action with the highest expected reward among action candidates under uncertainty [17]. Applications have been applied to several fields, for instance, recommendation systems [18], information retrieval [19], healthcare [20], etc. Several current pieces of research about the application of bandit-based approaches to the energy resource management problem are provided in Table 1.…”

“…Since the predicted load and demand response requirement have uncertainty, some DG, GT, and curtailable load capacities are spared to improve the reliability with a certain probability. The chance constraints are formulated in (19) and (20). δ load t , δ DR t are the corresponding random variables of the estimation errors and α is the risk parameter.…”

“…The CCP approach assumes that the estimation error follows a Gaussian distribution. Then, according to the reference [46], we transform (19) and (20) into the deterministic equivalents:…”