Onedata – A Step Forward towards Globalization of Data Access for Computing Infrastructures

Dutka, Łukasz; Wrzeszcz, Michał; Lichoń, Tomasz; Słota, Rafał; Zemek, Konrad; Trzepla, Krzysztof; Opioła, Łukasz; Słota, Renata; Kitowski, Jacek

doi:10.1016/j.procs.2015.05.445

Cited by 15 publications

(8 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The creation of an OSCAR function allows users to upload files to the object storage system which triggers the execution of the function to perform the data-processing, with automated elasticity if it is required, and the output data is stored in any of the object storage systems supported. This is the case of Onedata [29] a global data management system that provides access to distributed storage resources for data-intensive scientific computations. This is used to support EGI DataHub, a federated data storage layer auspiced by EGI (European Grid Infrastructure), the largest federated Cloud in Europe.…”

Section: Oscar: Open-source Serverless Computing For Data-processing Applicationsmentioning

confidence: 99%

Serverless Workflows for Containerised Applications in the Cloud Continuum

et al. 2021

View full text Add to dashboard Cite

This paper introduces an open-source platform to support serverless computing for scientific data-processing workflow-based applications across the Cloud continuum (i.e. simultaneously involving both on-premises and public Cloud platforms to process data captured at the edge). This is achieved via dynamic resource provisioning for FaaS platforms compatible with scale-to-zero approaches that minimise resource usage and cost for dynamic workloads with different elasticity requirements. The platform combines the usage of dynamically deployed auto-scaled Kubernetes clusters on on-premises Clouds and automated Cloud bursting into AWS Lambda to achieve higher levels of elasticity. A use case in public health for smart cities is used to assess the platform, in charge of detecting people not wearing face masks from captured videos. Faces are blurred for enhanced anonymity in the on-premises Cloud and detection via Deep Learning models is performed in AWS Lambda for this data-driven containerised workflow. The results indicate that hybrid workflows across the Cloud continuum can efficiently perform local data processing for enhanced regulations compliance and perform Cloud bursting for increased levels of elasticity.

show abstract

Section: Oscar: Open-source Serverless Computing For Data-processing Applicationsmentioning

confidence: 99%

Serverless Workflows for Containerised Applications in the Cloud Continuum

et al. 2021

View full text Add to dashboard Cite

show abstract

“…The data-access systems use metadata to describe various bits of information connected with data access; for example, user specific information (e.g., access control) and storage-specific information (e.g., location of data replicas) [14,64]. The metadata can also be used to describe the context of data access; e.g., the load of the data-access system components.…”

Section: Roadmapmentioning

confidence: 99%

Towards Transparent Data Access With Context Awareness

Wrzeszcz

Kitowski

Słota

2018

csci

Self Cite

View full text Add to dashboard Cite

Open-data research is an important factor accelerating the production and analysis of scientific results as well as worldwide collaboration; still, very little data is being shared at scale. The aim of this article is to analyze existing data-access solutions along with their usage limitations. After analyzing the existing solutions and data-access stakeholder needs, the authors propose their own vision of a data-access model.

show abstract

“…Advances in the acquisition and segmentation of high-throughput volume electron microscopy (VEM) create larger data sets (Kornfeld and Denk, 2018) that stress data management tools due to the volume of data, the need to support proofreading as well as automated, high-throughput batch operations, and the sharing and integration of results from different research groups. While many data distribution systems focus on large numbers of relatively small datasets or file-based distribution (Dutka et al, 2015; Viljoen et al, 2016), VEM reconstructions are not easily distributed and usable to researchers through file distribution. For teravoxel to petavoxel datasets, centralized data services can provide low latency access to areas of interest without requiring the download of much larger volumes of data (Saalfeld et al, 2009; Burns et al, 2013; Haehn et al, 2017; Kleissas et al, 2017).…”

Section: Introductionmentioning

confidence: 99%

DVID: Distributed Versioned Image-Oriented Dataservice

Katz

Plaza

2019

Front. Neural Circuits

View full text Add to dashboard Cite

Open-source software development has skyrocketed in part due to community tools like github.com, which allows publication of code as well as the ability to create branches and push accepted modifications back to the original repository. As the number and size of EM-based datasets increases, the connectomics community faces similar issues when we publish snapshot data corresponding to a publication. Ideally, there would be a mechanism where remote collaborators could modify branches of the data and then flexibly reintegrate results via moderated acceptance of changes. The DVID system provides a web-based connectomics API and the first steps toward such a distributed versioning approach to EM-based connectomics datasets. Through its use as the central data resource for Janelia's FlyEM team, we have integrated the concepts of distributed versioning into reconstruction workflows, allowing support for proofreader training and segmentation experiments through branched, versioned data. DVID also supports persistence to a variety of storage systems from high-speed local SSDs to cloud-based object stores, which allows its deployment on laptops as well as large servers. The tailoring of the backend storage to each type of connectomics data leads to efficient storage and fast queries. DVID is freely available as open-source software with an increasing number of supported storage options.

show abstract

Onedata – A Step Forward towards Globalization of Data Access for Computing Infrastructures

Cited by 15 publications

References 2 publications

Serverless Workflows for Containerised Applications in the Cloud Continuum

Serverless Workflows for Containerised Applications in the Cloud Continuum

Towards Transparent Data Access With Context Awareness

DVID: Distributed Versioned Image-Oriented Dataservice

Contact Info

Product

Resources

About