Arrow: Low-Level Augmented Bayesian Optimization for Finding the Best Cloud VM

Abstract

With the advent of big data applications, which tends to have longer execution time, choosing the right cloud VM to run these applications has significant performance as well as economic implications. For example, in our large-scale empirical study of 107 different workloads on three popular big data systems, we found that a wrong choice can lead to a 20 times slowdown or an increase in cost by 10 times.Bayesian optimization is a technique for optimizing expensive (black-box) functions. Previous attempts have only used instancelevel information (such as # of cores, memory size) which is not sufficient to represent the search space. In this work, we discover that this may lead to the fragility problem-either incurs high search cost or finds only the sub-optimal solution. The central insight of this paper is to use low-level performance information to augment the process of Bayesian Optimization. Our novel low-level augmented Bayesian Optimization is rarely worse than current practices and often performs much better (in 46 of 107 cases). Further, it significantly reduces the search cost in nearly half of our case studies.Based on this work, we conclude that it is often insufficient to use general-purpose off-the-shelf methods for configuring cloud instances without augmenting those methods with essential systems knowledge such as CPU utilization, working memory size and I/O wait time.

Micky: A Cheaper Alternative for Selecting Cloud Instances

“…We try our best to reflect the common cloud deployment. More details regrading data collection can be found in our previous work [5], [6]. We also made our data public available for further research [18].…”

“…Users are looking for configurations that are highly performing (e.g., the shortest execution time) or cost-effective (e.g., the cheapest operational cost), or meeting the trade-off between them [4], [1], [5]. A poor choice, for example, can lead to a 20 time slowdown or a 10 times increase in total cost [5]. Although cloud providers recommend the choice of VM types, it is too coarse grain to be effective [16], [21].…”

“…CherryPick uses Bayesian Optimization, which updates its beliefs (workload performance on configurations) and finds the best configuration sequentially [4]. Although Bayesian Optimization is powerful, it can be fragile when the search space is not well represented [5].…”

“…Arrow leverages low-level performance metrics to address the fragility issue in CherryPick due to insufficient representation in the search space and poor choices of the kernel function in Gaussian Process [5].…”

“…However, such optimization task is not straightforward due to opaque resource requirement [1], [2]. To address this challenge, prior work either builds prediction models (as in Ernest [3] and PARIS [1]) or uses sequential model-based optimization (as in CherryPick [4], Arrow [5] and Scout [6]).…”