A Trace-Based Performance Study of Autoscaling Workloads of Workflows in Datacenters

Abstract

To improve customer experience, datacenter operators offer support for simplifying application and resource management. For example, running workloads of workflows on behalf of customers is desirable, but requires increasingly more sophisticated autoscaling policies, that is, policies that dynamically provision resources for the customer. Although selecting and tuning autoscaling policies is a challenging task for datacenter operators, so far relatively few studies investigate the performance of autoscaling for workloads of workflows. Complementing previous knowledge, in this work we propose the first comprehensive performance study in the field. Using trace-based simulation, we compare state-of-the-art autoscaling policies across multiple application domains, workload arrival patterns (e.g., burstiness), and system utilization levels. We further investigate the interplay between autoscaling and regular allocation policies, and the complexity cost of autoscaling. Our quantitative study focuses not only on traditional performance metrics and on state-of-the-art elasticity metrics, but also on timeand memory-related autoscaling-complexity metrics. Our main results give strong and quantitative evidence about previously unreported operational behavior, for example, that autoscaling policies perform differently across application domains and by how much they differ.

An Experimental Evaluation of the Kubernetes Cluster Autoscaler in the Cloud

“…This is because both CA and CA-NAP do not scale-out base on CPU utilization threshold but rather do so whenever there are pods that could not be scheduled due to a shortage of resources. The large values for θ O are similar to those of some of the general-purpose autoscalers studied in [14] but different from those reported by [15], Chameleon [4] and Chamulteon [16], whereas almost all the above works except [15] report large values fo τ O similar to ours.…”

“…To enable comparison of novel autoscaling methods not only to static provisioning as done in the past, but also to other autoscaling algorithms, the SPEC Cloud Group developed a set of standard autoscaling performance metrics [7] that are now being used by a number of works for comparing multiple autoscalers. Ilyushkin et al [14] use these metrics to compare seven autoscaling policies from the state of the art, whereas Versluis et al [15] present a simulation-based experimental evaluation of autoscaling workloads of workflows in data centers. These works provide a better understanding of the performance of autoscaling policies proposed in the past decade.…”

“…Our work extends these works by using similar metrics to quantify the performance of the autoscaling policy of the Kubernetes platform which is popular in the cloud-native application development paradigm. However, [14] focuses on scientific workflows while [15] focuses on scientific, industrial and engineering workflows. In contrast, we focus on two representative containerized applications for Kubernetes.…”

The AtLarge Vision on the Design of Distributed Systems and Ecosystems

“…Although the in vitro experiments were useful, they could not address many questions related to diverse workloads and environments, because the former would have been too expensive to experiment with, and for the latter we did not have access to different environments (and of the right scale). We have therefore designed and conducted in silico, simulation-based experiments [128]. We found interesting discrepancies between the real-world software of the initial in vitro experiments and the software of the simulator, which we have developed independently; these discrepancies have allowed us to correct in time the real-world results, and emphasize the need for independent corroboration in the community [130].…”

“…When we started this work, there existed already several autoscaling approaches, but none had been evaluated comprehensively. Motivated also by the emergence of a new set of elasticity metrics, we have designed and performed several experiments [126], [127], [128].…”