Comprehensible Control for Researchers and Developers Facing Data Challenges

Abstract

The DARE platform enables researchers and their developers to exploit more capabilities to handle complexity and scale in data, computation and collaboration. Today's challenges pose increasing and urgent demands for this combination of capabilities. To meet technical, economic and governance constraints, application communities must use use shared digital infrastructure principally via virtualisation and mapping. This requires precise abstractions that retain their meaning while their implementations and infrastructures change. Giving specialists direct control over these capabilities with detail relevant to each discipline is necessary for adoption. Research agility, improved power and retained return on intellectual investment incentivise that adoption. We report on an architecture for establishing and sustaining the necessary optimised mappings and early evaluations of its feasibility with two application communities.

Active Provenance for Data-Intensive Workflows: Engaging Users and Developers

“…We start from the Waveform pre-processing workflow in Figure 4. It is composed of simple stateless PEs receiving and producing waveform time-series according to a data format defined in the Obspy software package 8 [28]. The provenance type SeismoType, listed in Table II, can handle such a payload thanks to the implementation of its extractItemMetadata method, which uses the Obspy toolkit to access seismic data and metadata.…”

“…It is in use for demanding applications in seismology and climate-impact research, for the VERCE 1 [7] and CLIPC 2 projects, and now the DARE project 3 . The DARE architecture and vision, described in a contemporary paper [8], is the context for our recent achievements, illustrated with the seismic rapid assessment use case. (Section VI).…”

DARE: A Reflective Platform Designed to Enable Agile Data-Driven Research on the Cloud

“…Examples of commercial clouds being used in largescale scientific contexts are found on both sides of the Atlantic: in the European Open Science Cloud 1 (EOSC) case as well as in the massive ongoing migration of data and other resources onto Amazon's AWS by NASA 2 . This work has been supported by the EU H2020 research and innovation programme under grant agreement No 777413. 1 https://ec.europa.eu/research/openscience/index.cfm?pg= open-science-cloud 2 https://aws.amazon.com/partners/success/nasa-image-library/ It follows that while potential for large scale data-driven experimentation increases, so does complexity. At the same time, making use of vendor-specific features may lead to lock-in.…”

“…6) Exposing all relevant functionality via a set of RESTful APIs that (1) effectively hide technical detail and (2) enable research developers to build solutions that exploit multiple underlying e-infrastructures with minimal effort. The overarching vision behind DARE as well as its main architectural considerations and components can be found in [2]. This paper describes the current technical instantiation in response to this vision, its main software components and their interactions.…”

“…As per its reference architecture [2], the DARE platform implements a logical knowledge-base as a series of stores and 18 https://rook.io/ 19 https://eudat.eu/services/b2drop registries. The semantification and tighter integration of these constituent stores and registries is ongoing work.…”