Energy-Aware Simulation of Workflow Execution in High Throughput Computing Systems

Abstract

Additional information: Use policyThe full-text may be used and/or reproduced, and given to third parties in any format or medium, without prior permission or charge, for personal research or study, educational, or not-for-prot purposes provided that:• a full bibliographic reference is made to the original source • a link is made to the metadata record in DRO • the full-text is not changed in any way The full-text must not be sold in any format or medium without the formal permission of the copyright holders.Please consult the full DRO policy for further details. Abstract-Workflows offer a great potential for enacting corelated jobs in an automated manner. This is especially desirable when workflows are large or there is a desire to run a workflow multiple times. Much research has been conducted in reducing the makespan of running workflows and maximising the utilisation of the resources they run on, with some existing research investigates how to reduce the energy consumption of workflows on dedicated resources. We extend the HTC-Sim simulation framework to support workflows allowing us to evaluate different scheduling strategies on the overheads and energy consumption of workflows run on non-dedicated systems. We evaluate a number of scheduling strategies from the literature in an environment where (workflow) jobs can be evicted by higher priority users.

HTC‐Sim: a trace‐driven simulation framework for energy consumption in high‐throughput computing systems

“…We shall extend our simulation environment to (i) include support MPI workloads spanning multiple compute nodes and (ii) incorporate support for workflow workloads, for example, by modelling the functioning of the Directed Acyclic Graph Manager , the meta‐scheduler used by HTCondor to handle the dependencies between workflow jobs. Our preliminary work in this area is presented in . This would allow the extension of existing Directed Acyclic Graph‐based application mapping techniques to be evaluated for energy consumption in the context of multi‐use clusters.…”

“…by modelling the functioning of the Directed Acyclic Graph Manager (DAGMan) [75], the metascheduler used by HTCondor to handle the dependencies between Workflow jobs. Our preliminary work in this area is presented in [76]. This would allow the extension of existing DAG-based application mapping techniques [77,78,79] to be evaluated for energy consumption in the context of multi-use clusters.…”

New advances in large‐scale distributed simulation and real‐time applications

“…The overlap between large distributed simulations and real‐time applications has been always present, and it is increasing fast recently because of the circumstances raised from the current new trends, such as Cloud computing , intelligent transportation systems , green computing , and virtual environments . As the overlap increases, new challenges appear, mostly for enabling applications, like serious gaming , virtual and augmented reality, and collaborative virtual environments , to execute seamlessly on distributed resources .…”

“…

The overlap between large distributed simulations and real-time applications [1][2][3] has been always present, and it is increasing fast recently because of the circumstances raised from the current new trends, such as Cloud computing [4,5], intelligent transportation systems [1,6], green computing [7,8], and virtual environments [2,9]. As the overlap increases, new challenges appear, mostly for enabling applications, like serious gaming [10], virtual and augmented reality, and collaborative virtual environments [11], to execute seamlessly on distributed resources [12][13][14].

This special issue intends to collect the state-of-the-art research works that include a set of concepts on high performance computing; design, modeling, and validation of distributed real-time systems; and multi-agent control and simulation systems.

…”