Characterizing, Modeling, and Accurately Simulating Power and Energy Consumption of I/O-intensive Scientific Workflows

Silva, Rafael Ferreira da; Casanova, Henri; Orgerie, Anne‐Cécile; Tanaka, Ryan; Deelman, Ewa; Suter, Frédéric

doi:10.1016/j.jocs.2020.101157

Cited by 16 publications

(10 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Three main components contribute to the energy consumption in the Edge-DCs: the compute nodes, the switches and the cooling system. The power consumption of a compute node is made up of two terms: a static consumption that accounts for the power consumed when the server is powered on but not running any load or running background activities, and a dynamic consumption that depends linearly on the CPU frequency and the nature of the compute load [30]. Adelin et al [31] showed that a switch consumes more than 80% of its nominal power when it is switched on, including when there is no traffic (static state).…”

Section: A Edge-data Centers Power Consumptionmentioning

confidence: 99%

Latency, Energy and Carbon Aware Collaborative Resource Allocation with Consolidation and QoS Degradation Strategies in Edge Computing

Gnibga,

Blavette,

Orgerie

2023

2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS)

Self Cite

View full text Add to dashboard Cite

Edge Computing has emerged from the Cloud to tackle the increasingly stringent latency, reliability and scalability imperatives of modern applications, mainly in the Internet of Things arena. To this end, the data centers are pushed to the edge of the network to diversify and bring the services closer to the users. This spatial distribution offer a wide range of opportunities for allowing self-consumption from local renewable energy sources with regard to the local weather conditions. However, scheduling the users' tasks so as to meet the service restrictions while consuming the most renewable energy and reducing the carbon footprint remains a challenge. In this paper, we design a nationwide Edge infrastructure, and study its behavior under three typical electrical configurations including solar power plant, batteries and the grid. Then, we study a set of techniques that collaboratively allocates resources on the edge data centers to harvest renewable energy and reduce the environmental impact. These strategies also includes energy efficiency optimization by means of reasonable quality of service degradation and consolidation techniques at each data center in order to reduce the need for brown energy. The simulation results show that combining these techniques allows to increase the selfconsumption of the platform by 7.83% and to reduce the carbon footprint by 35.7% compared to the baseline algorithm. The optimizations also outperform classical energy-aware resource management algorithms from the literature. Yet, these techniques do not equally contribute to these performances, consolidation being the most efficient.

show abstract

Section: A Edge-data Centers Power Consumptionmentioning

confidence: 99%

Latency, Energy and Carbon Aware Collaborative Resource Allocation with Consolidation and QoS Degradation Strategies in Edge Computing

Gnibga,

Blavette,

Orgerie

2023

2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In [40], [41], we have proposed and validated a power consumption model that accounts for CPU utilization, computations that execute on multi-core compute nodes, and I/O operations (including the idle power consumption caused by waiting for these operations to complete). In this work, we leverage this model, to estimate the energy consumption of the execution of large-scale workflows.…”

Section: Case Study: Estimating Energy Consumption Of Large-scale Wor...mentioning

confidence: 99%

WfCommons: A Framework for Enabling Scientific Workflow Research and Development

Coleman¹,

Casanova²,

Pottier³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Scientific workflows are a cornerstone of modern scientific computing. They are used to describe complex computational applications that require efficient and robust management of large volumes of data, which are typically stored/processed on heterogeneous, distributed resources. The workflow research and development community has employed a number of methods for the quantitative evaluation of existing and novel workflow algorithms and systems. In particular, a common approach is to simulate workflow executions. In previous works, we have presented a collection of tools that have been adopted by the community for conducting workflow research. Despite their popularity, they suffer from several shortcomings that prevent easy adoption, maintenance, and consistency with the evolving structures and computational requirements of production workflows. In this work, we present WfCommons, a framework that provides a collection of tools for analyzing workflow executions, for producing generators of synthetic workflows, and for simulating workflow executions. We demonstrate the realism of the generated synthetic workflows by comparing their simulated executions to real workflow executions. We also contrast these results with results obtained when using the previously available collection of tools. We find that the workflow generators that are automatically constructed by our framework not only generate representative same-scale workflows (i.e., with structures and task characteristics distributions that resemble those observed in realworld workflows), but also do so at scales larger than that of available real-world workflows. Finally, we conduct a case study to demonstrate the usefulness of our framework for estimating the energy consumption of large-scale workflow executions.

show abstract

“…When such workflows are executed over large amounts of data, their runtimes can easily exceed days or even weeks [2,8,26,27]. To effectively use the available cluster resources, many workflow management systems have a scheduling component [6,12,24,31] that determines which tasks are executed when and on which of the available nodes to achieve some optimization goal, such as wallclock-time.…”

Section: Introductionmentioning

confidence: 99%

Lotaru: Locally Estimating Runtimes of Scientific Workflow Tasks in Heterogeneous Clusters

Bader,

Lehmann,

Thamsen

et al. 2022

Preprint

View full text Add to dashboard Cite

Many scientific workflow scheduling algorithms need to be informed about task runtimes a-priori to conduct efficient scheduling. In heterogeneous cluster infrastructures, this problem becomes aggravated because these runtimes are required for each task-node pair. Using historical data is often not feasible as logs are typically not retained indefinitely and workloads as well as infrastructure changes. In contrast, online methods, which predict task runtimes on specific nodes while the workflow is running, have to cope with the lack of example runs, especially during the start-up.In this paper, we present Lotaru, a novel online method for locally estimating task runtimes in scientific workflows on heterogeneous clusters. Lotaru first profiles all nodes of a cluster with a set of shortrunning and uniform microbenchmarks. Next, it runs the workflow to be scheduled on the user's local machine with drastically reduced data to determine important task characteristics. Based on these measurements, Lotaru learns a Bayesian linear regression model to predict a task's runtime given the input size and finally adjusts the predicted runtime specifically for each task-node pair in the cluster based on the micro-benchmark results. Due to its Bayesian approach, Lotaru can also compute robust uncertainty estimates and provides them as an input for advanced scheduling methods.Our evaluation with five real-world scientific workflows and different datasets shows that Lotaru significantly outperforms the baselines in terms of prediction errors for homogeneous and heterogeneous clusters. CCS CONCEPTS• Information systems → Information systems applications; • Computer systems organization → Distributed architectures; • Software and its engineering → Software architectures.

show abstract

Characterizing, Modeling, and Accurately Simulating Power and Energy Consumption of I/O-intensive Scientific Workflows

Cited by 16 publications

References 41 publications

Latency, Energy and Carbon Aware Collaborative Resource Allocation with Consolidation and QoS Degradation Strategies in Edge Computing

Latency, Energy and Carbon Aware Collaborative Resource Allocation with Consolidation and QoS Degradation Strategies in Edge Computing

WfCommons: A Framework for Enabling Scientific Workflow Research and Development

Lotaru: Locally Estimating Runtimes of Scientific Workflow Tasks in Heterogeneous Clusters

Contact Info

Product

Resources

About