Accurately Simulating Energy Consumption of I/O-Intensive Scientific Workflows

Silva, Rafael Ferreira da; Orgerie, Anne‐Cécile; Casanova, Henri; Tanaka, Ryan; Deelman, Ewa; Suter, Frédéric

doi:10.1007/978-3-030-22734-0_11

Cited by 9 publications

(4 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [40], [41], we have proposed and validated a power consumption model that accounts for CPU utilization, computations that execute on multi-core compute nodes, and I/O operations (including the idle power consumption caused by waiting for these operations to complete). In this work, we leverage this model, to estimate the energy consumption of the execution of large-scale workflows.…”

Section: Case Study: Estimating Energy Consumption Of Large-scale Wor...mentioning

confidence: 99%

WfCommons: A Framework for Enabling Scientific Workflow Research and Development

Coleman¹,

Casanova²,

Pottier³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Scientific workflows are a cornerstone of modern scientific computing. They are used to describe complex computational applications that require efficient and robust management of large volumes of data, which are typically stored/processed on heterogeneous, distributed resources. The workflow research and development community has employed a number of methods for the quantitative evaluation of existing and novel workflow algorithms and systems. In particular, a common approach is to simulate workflow executions. In previous works, we have presented a collection of tools that have been adopted by the community for conducting workflow research. Despite their popularity, they suffer from several shortcomings that prevent easy adoption, maintenance, and consistency with the evolving structures and computational requirements of production workflows. In this work, we present WfCommons, a framework that provides a collection of tools for analyzing workflow executions, for producing generators of synthetic workflows, and for simulating workflow executions. We demonstrate the realism of the generated synthetic workflows by comparing their simulated executions to real workflow executions. We also contrast these results with results obtained when using the previously available collection of tools. We find that the workflow generators that are automatically constructed by our framework not only generate representative same-scale workflows (i.e., with structures and task characteristics distributions that resemble those observed in realworld workflows), but also do so at scales larger than that of available real-world workflows. Finally, we conduct a case study to demonstrate the usefulness of our framework for estimating the energy consumption of large-scale workflow executions.

show abstract

Section: Case Study: Estimating Energy Consumption Of Large-scale Wor...mentioning

confidence: 99%

WfCommons: A Framework for Enabling Scientific Workflow Research and Development

Coleman¹,

Casanova²,

Pottier³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Grid'5000 is a testbed for experiment-driven research, which provides resource isolation and advanced monitoring and measurement features for the collection of power consumption traces. 1 A preliminary version of this paper appears in the proceedings of the 2019 International Conference on Computational Science (ICCS) [16].…”

Section: Workflow Characterizationmentioning

confidence: 99%

Characterizing, Modeling, and Accurately Simulating Power and Energy Consumption of I/O-intensive Scientific Workflows

Silva

Casanova

Orgerie

et al. 2020

Journal of Computational Science

Self Cite

View full text Add to dashboard Cite

While distributed computing infrastructures can provide infrastructure-level techniques for managing energy consumption, application-level energy consumption models have also been developed to support energy-efficient scheduling and resource provisioning algorithms. In this work, we analyze the accuracy of a widely-used application-level model that has been developed and used in the context of scientific workflow executions. To this end, we profile two production scientific workflows on a distributed platform instrumented with power meters. We then conduct an analysis of power and energy consumption measurements. This analysis shows that power consumption is not linearly related to CPU utilization and that I/O operations significantly impact power, and thus energy, consumption. We then propose a power consumption model that accounts for I/O operations, including the impact of waiting for these operations to complete, and for concurrent task executions on multi-socket, multi-core compute nodes. We implement our proposed model as part of a simulator that allows us to draw direct comparisons between real-world and modeled power and energy consumption. We find that our model has high accuracy when compared to real-world executions. Furthermore, our model improves accuracy by about two orders of magnitude when compared to the traditional models used in the energy-efficient workflow scheduling literature.

show abstract

“…WRENCH has already enabled the development of two simulators of two production application execution frameworks [8], [9] for supporting the study of energy-efficiency for I/O-intensive workflow applications [10], and the study of simulation accuracy and scalability [2]. WRENCH has also gained overseas attention, and received contributions from users in France and Ivory Coast, who are conducting research on workflow scheduling.…”

Section: Wrench's Impact On CI Research Development and Educationmentioning

confidence: 99%

“…In a recent work [10], we have leveraged the WRENCHenabled Pegasus simulator to investigate the impact of resource utilization and I/O operations on the energy usage, as well as the impact of executing multiple tasks concurrently on multisocket, multi-core compute nodes. Our simulator allowed us to draw direct comparisons between real-world and modeled power and energy consumption.…”

Section: B Energy-aware Computingmentioning

confidence: 99%

Bridging Concepts and Practice in eScience via Simulation-Driven Engineering

Silva

Casanova

Tanaka

et al. 2019

2019 15th International Conference on eScience (eScience)

Self Cite

View full text Add to dashboard Cite

The CyberInfrastructure (CI) has been the object of intensive research and development in the last decade, resulting in a rich set of abstractions and interoperable software implementations that are used in production today for supporting ongoing and breakthrough scientific discoveries. A key challenge is the development of tools and application execution frameworks that are robust in current and emerging CI configurations, and that can anticipate the needs of upcoming CI applications. This paper presents WRENCH, a framework that enables simulationdriven engineering for evaluating and developing CI application execution frameworks. WRENCH provides a set of highlevel simulation abstractions that serve as building blocks for developing custom simulators. These abstractions rely on the scalable and accurate simulation models that are provided by the SimGrid simulation framework. Consequently, WRENCH makes it possible to build, with minimum software development effort, simulators that that can accurately and scalably simulate a wide spectrum of large and complex CI scenarios. These simulators can then be used to evaluate and/or compare alternate platform, system, and algorithm designs, so as to drive the development of CI solutions for current and emerging applications.

show abstract

Accurately Simulating Energy Consumption of I/O-Intensive Scientific Workflows

Cited by 9 publications

References 27 publications

WfCommons: A Framework for Enabling Scientific Workflow Research and Development

WfCommons: A Framework for Enabling Scientific Workflow Research and Development

Characterizing, Modeling, and Accurately Simulating Power and Energy Consumption of I/O-intensive Scientific Workflows

Bridging Concepts and Practice in eScience via Simulation-Driven Engineering

Contact Info

Product

Resources

About