Towards Addressing CPU-Intensive Seismological Applications in Europe

Carpené, Michele; Klampanos, Iraklis A.; Leong, Siew Hoon; Casarotti, Emanuele; Danecek, Peter; Ferini, G.; Gemünd, André; Krause, Amrey; Krischer, Lion; Magnoni, Federica; Simon, Marek Sebastian; Spinuso, Alessandro; Trani, Luca; Atkinson, Malcolm; Erbacci, Giovanni; Frank, Anton; Igel, Heiner; Rietbrock, Andreas; Schwichtenberg, Horst; Vilotte, Jean‐Pierre

doi:10.1007/978-3-642-38750-0_5

Cited by 2 publications

(1 citation statement)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In principle, at least, the units passed in a data stream can be arbitrarily large, e.g. multi-dimensional arrays denoting successive states in a finite element model (FEM) of a dynamic system [43], though they are typically time-stamped tuples often encoding time and a small number of scalars in a highly compressed form. The data-streaming model can also pass file names in its data units and thereby draw on the facilities task-oriented systems use.…”

Section: Introductionmentioning

confidence: 99%

`dispel4py`: A Python framework for data-intensive scientific computing

Filguiera

Krause

Atkinson

et al. 2016

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

This paper presents dispel4py, a new Python framework for describing abstract stream-based workflows for distributed data-intensive applications. These combine the familiarity of Python programming with the scalability of workflows. Data streaming is used to gain performance, rapid prototyping and applicability to live observations. dispel4py enables scientists to focus on their scientific goals, avoiding distracting details and retaining flexibility over the computing infrastructure they use. The implementation, therefore, has to map dispel4py abstract workflows optimally onto target platforms chosen dynamically. We present four dispel4py mappings: Apache Storm, MPI, multi-threading and sequential, showing two major benefits: a) smooth transitions from local development on a laptop to scalable execution for production work, and b) scalable enactment on significantly different distributed computing infrastructures. Three application domains are reported and measurements on multiple infrastructures show the optimisations achieved; they have provided demanding real applications and helped us develop effective training. The dispel4py.org is an open-source project to which we invite participation. The effective mapping of dispel4py onto multiple target infrastructures demonstrates exploitation of data-intensive and HPC architectures and consistent scalability.

show abstract

Section: Introductionmentioning

confidence: 99%