Simulating Application Workflows and Services Deployed on the European Grid Infrastructure

Abstract

International audienceThis paper presents an end-to-end SimGrid-based simulation of a Monte-Carlo computation deployed on the European Grid Infrastructure. We describe a simulation framework allowing to replay executions from real traces. Middleware services, namely the file catalog, storage elements, the DIRAC pilot-job system, and the MOTEUR workflow engine are simulated by SimGrid processes. The deployment of pilot jobs, performed on EGI by the gLite WMS and batch queues, is simulated by a random selection of platform hosts, and a matching of their latencies and failure times with real traces. The Monte-Carlo application workflow is calibrated to address performance discrepancies between the real and simulated network and CPUs. The simulation is evaluated against real executions of the GATE Monte-Carlo application. Results show that SimGrid can be used to study the performance of applications running on EGI. Simulated and real make spans are consistent, and conclusions drawn in production about the influence of application parameters such as the check pointing frequency and the number of mergers are also made in simulation. This opens the door to better and faster experimentation on production grids

Adding Storage Simulation Capacities to the SimGrid Toolkit: Concepts, Models, and API

“…To be valuable, the conclusions drawn by such simulations studies have to remain sound once the developed solutions are deployed in production. The level of realism offered by SimGrid in terms of network simulation has already been leveraged in [20]. Adding the simulation of high-level storage components such as file catalog, storage resource manager, or gridFTP services, will reinforce this realism.…”

“…Such studies leveraging SimGrid have been published before [7], [20], [21], but the lack of storage simulation capacities forced their authors to either ignore, circumvent, or implement in user space the storage part of their simulators. In the remaining of this section we present different envisioned use cases that could benefit of the proposed extension as is, or require alternate models of storage resources.…”

“…Then, the first instance, named Disk1 (lines [14][15], which is attached to bob does not specify any specific list of files and thus inherits of the generic list of its type. Conversely the second instance, named Disk2 and attached to alice (lines [17][18][19][20], is given a specific list of file, described in a different format, that of Windows TM . The method used to parse file lists derives from the value of the content_type attribute.…”

Community Resources for Enabling Research in Distributed Scientific Workflows

“…Several workflow simulator frameworks [2], [39] have been developed to address a single characteristic of the WMS such as workflow scheduling. CloudSim [3] is a general-purpose framework for modeling and simulating cloud computing infrastructures and services, however it only supports the execution of single task workloads.…”

High Performance Scientific Computing over Hybrid Cloud Platforms

“…This requires a significant amount of computing power and memory capacity that can be only obtained via distributed computing. Indeed, large-scale Distributed Computing Infrastructures (DCIs), such as the EGI 1 have been tremendously successful in supporting the computational requirements of many scientific communities across Europe [183,43]. However, one of the main limitations of Grid infrastructures is that applications have to be ported to the execution environments provided by the machines involved, what results in a rigid structure composed by several VOs that support a set of applications.…”