An automated approach to create, store, and analyze large-scale experimental data in clouds

Abstract

Abstract-The flexibility and scalability of computing clouds make them an attractive application migration target; yet, the cloud remains a black-box for the most part. In particular, their opacity impedes the efficient but necessary testing and tuning prior to moving new applications into the cloud. A natural and presumably unbiased approach to reveal the cloud's complexity is to collect significant performance data by conducting more experimental studies. However, conducting large-scale system experiments is particularly challenging because of the practical difficulties that arise during experimental deployment, configuration, execution and data processing. In this paper we address some of these challenges through Expertus -a flexible automation framework we have developed to create, store and analyze large-scale experimental measurement data. We create performance data by automating the measurement processes for large-scale experimentation, including: the application deployment, configuration, workload execution and data collection processes. We have automated the processing of heterogeneous data as well as the storage of it in a data warehouse, which we have specifically designed for housing measurement data. Finally, we have developed a rich web portal to navigate, statistically analyze and visualize the collected data. Expertus combines template-driven code generation techniques with aspect-oriented programming concepts to generate the necessary resources to fully automate the experiment measurement process. In Expertus, a researcher provides only the high-level description about the experiment, and the framework does everything else. At the end, the researcher can graphically navigate and process the data in the web portal.

Cloud Crawler: a declarative performance evaluation environment for infrastructure‐as‐a‐service clouds

“…In addition to Cloud Crawler, described in this paper, other related tools that follow this approach are Expertus [16], CloudBench [17], and Cloud WorkBench [18]). …”

“…The first part of the specification (lines 1-43) defines the global entities of the WordPress benchmark, including three VMs (lines 9-32) that will run two application components (i.e., MySQL and Nginx) and the load generator (Gatling); one workload (lines [34][35][36][37][38] containing the selected number of WordPress users to be created at each test by Gatling (note that the Gatling VM is declared as the workload's target attribute); and one configuration script (lines 40-43) that will be used to provide the Apache server with the IP address of the MySQL database VM. Note also how the Gatling VM is associated with a configuration script named submit (lines [16][17][18], and the MySQL VM is associated with an unnamed shutdown script (lines 25-28). The Gatling script will be responsible for providing Gatling with the dynamic IP address of the Nginx VM, which is referred to in Crawl as the entity named driver (declared in lines 61 and 80).…”

“…This approach comprises automated benchmarking tools that can be used to systematically evaluate the performance of a given cloud application in one or more target cloud providers. In addition to Cloud Crawler, described in this paper, other related tools that follow this approach are Expertus [16], CloudBench [17], and Cloud WorkBench [18]).…”

“…In addition, depending on the duration of each test, and on the number of cloud providers and/or resource configurations being tested, the total cost of running the tests can be difficult to predict, demanding that cloud users have a careful control over their resource usage.In this paper, we present Cloud Crawler, a new automated environment aimed at facilitating both the specification and the execution of application performance tests in multiple IaaS clouds. In contrast to the more imperative (i.e., programatic) approaches followed by other automated solutions with similar goals, such as Expertus [16], CloudBench [17], and Cloud WorkBench [18], with Cloud Crawler, cloud users specify the performance tests to be executed in the cloud declaratively, at a higher abstraction level. To this end, the environment provides a novel declarative language, called Crawl, which allows the description of a rich variety of performance evaluation scenarios for a given cloud application.…”

Cloud WorkBench

“…Moreover, in fast moving cloud environments, continuous reevaluation is inevitable, when providers change their supported instance types or upgrade their hardware. Therefore, several research projects [1,5,9] aiming at extensible cloud benchmark automation were recently introduced. They all facilitate systematic cloud benchmarking.…”