Remote storage management in science gateways via data bridging

Abstract

SUMMARYState-of-the-art science gateways can be connected to several distributed computing infrastructures (DCIs) and are able to run jobs and workflows simultaneously in all those DCIs. Flexibility of accessing diverse data storages from these workflows and assisting end users to manage these storages are however the missing features in current gateway implementations, in which these problems often prove to be a barrier of exploiting the power of distributed computing by user communities having no or little IT competence. This paper addresses these issues by integrating a data bridging service called Data Avenue into WS-PGRADE/gUSE portal framework. Data Avenue offers tools for the end users to easily manage their data residing on various storage resources, and also, jobs become capable of accessing different storages regardless of the particular distributed computing infrastructure where the job is currently being run.

Data Migration for Large Scientific Datasets in Clouds

“…Data Avenue was originally motivated to allow grid applications to access diverse storage resources from different distributed computing infrastructures (DCIs) without the need of installing additional software, or using proprietary, storage-specific data transfer tools in application codes respectively [4], Previously, distributed applications, which were willing to read/store greater amount of data, must had been adapted at code-level to the particular storage provided in that particular DCI (e.g. access to the specific GridFTP site was part of application code using specific libraries deployed in all the worker nodes of the DCI).…”

“…From these results we derived the transfer rate value (MB/s) for each granularity, file size. We used curl [4] (version 7.47.0) to communicate with Data Avenue REST API service, AWS CLI (version 1.14.7) in order to connect to the S3 storage directly. In our experiments all the hosts: S3 storage, Data Avenue server, client machine, were virtual machines resided in the same cloud (MTA Cloud).…”

ENTICE VM Image Analysis and Optimised Fragmentation

“…As base image we used an image based on the cloud image provided by Canonical based on Ubuntu 16.04 LTS [8]. On top of the LTM stack we deployed the Data Avenue [16] application. Table 2 presents the results.…”

Science gateway workshops 2014 special issue conference publications

“…Gateways again can provide a natural interface to managing data effectively. Our first submission, ‘Remote Storage Management in Science Gateways via Data Bridging’ actually connects challenges in data management with the challenges of workflows and distributed computing infrastructures described in the preceding section. A data bridging service called Data Avenue has been integrated into the WS‐PGRADE/gUSE portal framework to provide a common interface to the myriad storage resources, thus streamlining the use of workflows that use many different underlying infrastructures.…”