DARE: A Reflective Platform Designed to Enable Agile Data-Driven Research on the Cloud

Abstract

The DARE platform has been designed to help research developers deliver user-facing applications and solutions over diverse underlying e-infrastructures, data and computational contexts. The platform is Cloud-ready, and relies on the exposure of API, which are suitable for raising the abstraction level and hiding complexity. It implements the cataloguing and execution of fine-grained and Python-based dispel4py workflows as services. Reflection is achieved via a logical knowledge base, comprising multiple internal catalogues, registries and semantics, while it supports persistent and pervasive data provenance. This paper presents design and implementation aspects of the DARE platform, as well as it provides directions for future development.

Comprehensible Control for Researchers and Developers Facing Data Challenges

“…Virtualisation and containerisation is key to feasibility and sustainability of the framework. There is currently a prototype with sophisticated versions of WaaS and P4, and a preliminary version of DKB [18]. This is already used by two application communities.…”

“…III) is prototyping a sociotechnical strategy for achieving and sustaining long-term and challenging objectives. The current implementation [18], outlined below, is a significant step towards those objectives that supports our two use cases.…”

Supercomputing with MPI meets the Common Workflow Language standards: an experience report

“…There are other, arguably less tightly coupled, general purpose approaches that support the expression of the stages through configuration files. A number of these, including DARE [10] and Pegasus [11], support execution of their workflows on HPC machines. For instance a specific tool, pegasus-mpi-cluster [12], has been developed for Pegasus which enables it to run highthroughput scientific workflows on HPC systems.…”

“…We show descriptions of the allowed keys, their types, and their default values if omitted in table I. 10 The Within the runner, this argument, if present, is used to configure the MPI runtime. When a tool is actually executed, the runner checks for the MPIRequirement, evaluates the processes attribute and if present and non-zero, it uses the configuration data, as described in table I, to prepend the appropriate strings to the front of the command line and alter the runtime environment.…”

Adaptive Optimizations for Stream-based Workflows

“…Therefore, many streambased workflow systems have been implemented for solving diverse objectives, including dispel4py [5]. dispel4py is a python library for data-intensive processing which has been well-developed and gained recognition of many scientists from different disciplines varying from seismology to astronomy [6], [7]. It offers mappings to several enactment engines, such as MPI [8], Storm [9], or multiprocessing 1 , and provides smooths transitions from local development to scalable executions.…”